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ABSTRACT 

...  The  purpose  of  this  research  is  to  ascertain  if  the  United  States  Air  Force  (USAF)  is 
duplicating  effort  with  the  development  of  the  two  IT  applications,  the  Warrior  Support 
Tool  (WST)  and  the  Agile  Munitions  Support  Tool  (AMST),  or  if  one  will  effectively 
meet  requirements.  Specifically,  this  thesis  sought  to  answer  four  research  questions 
addressing  customer  needs  for  a  munitions  IT  application,  identifying  the  capabilities  of 
the  two  research  IT  applications,  determining  how  well  each  application  met  the  needs  of 
the  customer,  and  detennining  how  supportable  each  application  was  in  terms  of 
information  needs  versus  existing  munitions  systems.  A  mixed  qualitative  and 
quantitative  paradigm  was  used  to  conduct  a  gap  analysis  and  verification  occurred 
through  the  use  of  surveys.  Sixty-five  munitions  users  participated  in  a  surveying  session 
to  identify  differences  between  the  two  systems  as  well  as  the  relative  merits  of  each. 
Overall,  munitions  users  chose  WST  as  the  more  favorable  application.  All  but  two 
construct  measures  supported  the  overall  rating  of  the  application.  Munitions  personnel 
liked  the  screen  layout  of  AMST  better  than  that  of  WST  even  though  WST  seemed  to  be 
the  application  that  better  used  terminology  familiar  to  munitions  personnel.  WST  was 
also  the  easier  of  the  two  applications  to  manipulate  although  the  AMST  help  features 
were  identified  as  the  better  of  the  two.  The  Warrior  Support  Tool  provided  infonnation 
in  a  timelier  manner  than  did  the  Agile  Munitions  Support  Tool. 
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A  COMPARATIVE  STUDY  OF  THE  WARRIOR  SUPPORT  TOOL  AND  THE 


AGILE  MUNITIONS  SUPPORT  TOOL 


I.  Introduction 


Overview 

The  way  the  United  States  military  plans  to  fight  and  support  the  war  fighter  has 
changed  in  recent  years.  The  logistics  community  was  tasked  to  provide  Focused 
Logistics.  Department  of  Defense  (DoD)  Joint  Vision  (JV)  2020  states  that 

Focused  Logistics  is  the  ability  to  provide  the  joint  force  the  right 
personnel,  equipment,  and  supplies  in  the  right  place,  at  the  right  time,  and 
in  the  right  quantity,  across  the  full  range  of  military  operations.  This  will 
be  made  possible  through  a  real-time,  web-based  information  system. . . 

(DoD  JV  2020,  2000) 

However,  very  little  has  changed  in  the  way  we  gather  information  to  improve 
that  support.  The  1990s  began  a  surge  in  the  growth  of  information  technology.  Many 
creative  ways  of  collecting  and  managing  infonnation  were  established  to  provide  useful 
data  to  decision  makers  to  improve  the  decision  making  process.  Currently,  a  lot  of 
munitions  data  is  collected  and  documented,  but  it  is  scattered  through  many  disparate 
databases  and  is  often  untimely.  A  great  deal  of  time  and  effort  is  spent  manually 
retrieving,  compiling,  and  analyzing  data  to  provide  meaningful  infonnation.  There 
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exists  a  need  for  an  Information  Technology  (IT)  application  that  can  automatically  pull 
munitions  data  from  ammunition  information  systems  and  combine  that  data  to  paint  a 
complete  picture  for  the  user.  Such  a  technology  would  provide  commanders  and 
munitions  users  the  tools  and  information  necessary  to  make  the  timely  and  proactive 
decisions  necessary  to  keep  our  United  States  military  poised  to  respond  to  world  crises. 
The  purpose  of  this  research  is  to  investigate  the  ability  of  two  information  technology 
systems  to  meet  this  need. 

Background 

In  order  to  make  accurate  munitions  decisions,  one  must  understand  the  status  of 
the  munitions  inventory,  what  weapons  are  being  produced,  what  weapons  are  planned 
for  production,  how  expenditures  rates  may  impact  planning,  and  how  future  weapons 
could  replace  current  assets  by  providing  increased  capability.  This  infonnation  can  only 
be  properly  analyzed  through  the  use  of  a  decision  support  tool.  Decision  support  tools 
come  in  many  forms  and  are  distinguished  by  the  level  of  interaction  with  the  user.  The 
Logistics  Control  and  Information  Document  states  that  low-interaction  support  tools 
require  a  lot  of  inputs  and  manipulations  of  mathematical  equations  from  the  developer  to 
enable  the  tool  to  perform  complex  processes  in  minimum  time.  High-interaction  tools 
rely  on  the  user  to  provide  processing  information  (Paperless  Acquisition  Initiative, 

1997).  An  application  that  displays  all  information  relevant  in  a  decision  process  allows 
the  user  to  detennine  what  is  important  or  not.  Two  applications,  the  Warrior  Support 
Tool  (WST)  and  the  Agile  Munitions  Support  Tool  (AMST)  have,  for  the  first  time, 
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proposed  to  provide  munitions  planners  the  ability  to  view  and  understand  these 
interrelationships  within  an  integrated  application. 

Rob  Roy,  president  of  Decision  Sciences  Incorporated  (DSI),  states  in  Warrior 
Support  Tool  literature  that  “WST  currently  supports  the  mission  of  the  Air  Armament 
Center  (AAC)  by  providing  enterprise  level  visibility  of  critical  munitions  within  a  web- 
based  environment”  (Rob  Roy,  July  11,  2002).  AAC,  the  Air  Force’s  test  and 
development  center  for  munitions  assets,  is  located  on  Eglin  Air  Force  Base  (AFB),  in  Ft. 
Walton  Beach,  Florida.  A  DSI  storyboard  presentation  states  “WST  provides  display  of 
inventory  availability,  munitions  status,  on-going  and  planned  acquisition,  and  allows  for 
predictive  analysis  of  future  inventory  based  on  current  planning  and  industrial  base 
considerations”  (Rob  Roy,  July  11,  2002). 

Similarly,  literature  from  Synergy  Incorporated  states  that  “AMST  is  a  web- 
enabled  application  with  an  Oracle  database  that  facilitates  access  to  Combat 
Ammunition  System  (CAS)  Ammunition  Control  Point/Air  Logistics  Center  (CAS-A) 
data  and  allows  different  echelons  of  munitions  management  to  query,  extract,  and 
analyze  asset  balance  data”  (Synergy  Inc.,  2002).  AMST  is  currently  in  use  at  Hill  AFB, 
the  Air  Force  (AF)  depot  for  air-to-ground  munitions.  The  Synergy  user  guide  states, 
“AMST  will  interface  with  Anny  Depot  Systems,  the  Global  Transportation  Network 
(GTN),  and  other  systems  necessary  to  provide  maximum  visibility  of  all  Air  Force 
munitions  assets  and  in-transits”  (Synergy  Inc.,  2002). 

Near  real-time  information  updates  from  official  sources  such  as  CAS  are 
integrated  by  both  applications  to  provide  munitions  planners  the  ability  to  make  timely 
decisions  on  how  to  manage  munitions  assets.  Rob  Roy  states,  “The  WST  application  is 
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comprised  of  the  integration  of  system  software,  system  hardware,  and  personal  computer 
displays  to  provide  a  tool  for  munitions  tracking,  predictions,  identification  of  platfonn 
compatibility,  and  responsive  capabilities  of  accelerated  production  for  acquisition 
planning”  (July  11,  2002).  The  AMST  User  Guide  states: 


The  AMST  application  is  hosted  on  a  dedicated  Dell  Power  Edge  440 
Server  running  the  Windows  NT  (Service  Pack  5)  operating  system.  The 
Oracle  8i  database  is  hosted  on  a  dedicated  Sun  Enterprise  450  Server  and 
runs  under  the  Sun  Server  (Version  8)  operating  system  and  operates  in 
the  System  High  Mode  at  the  UNCLASSIFIED-SENSITIVE  level  and 
employs  Class  C2,  Controlled  Access  Protection,  and  Trusted  Computing 
Base  security  protection  features  and  functionality  configured  in 
accordance  with  Department  of  Defense  5200.28-STD,  AFMAN  33-229, 
and  AFSSI  5024,  volume  1.  (2002) 

WST  and  AMST  both  appear  to  have  the  capability  to  provide  the  infonnation 
needed  to  support  munitions  operations.  This  study  will  investigate  the  applications  in 
detail  to  determine  how  well  they  meet  munitions  needs. 


Problem  Statement 

All  of  the  pertinent  information  for  good  weapon  planning  is  available  in  the 
munitions  legacy  systems;  however,  the  information  is  viewed  in  a  piecemeal  and 
disjointed  fashion.  Munitions  planners  and  users  need  a  source  for  real-time  munitions 
information  that  can  provide  an  accurate  picture  of  munitions  capabilities.  The 
information  needs  to  be  integrated  and  presented  in  the  proper  format  for  decision¬ 
making. 
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Research  Question 


This  investigation  seeks  to  address  the  problem  by  answering  two  related  research 
questions.  First,  how  well  do  WST  and  AMST  meet  the  decision  support  needs  of  the 
retail  munitions  community?  Also,  what  limitations  or  shortcomings  are  present  in  the 
two  models? 

Investigative  Questions 

Decision  makers  need  to  know  the  strengths  and  weaknesses  of  the  Warrior 
Support  Tool  versus  the  Agile  Munitions  Support  Tool.  The  purpose  of  this  research  is  to 
ascertain  if  the  USAF  is  duplicating  effort  with  the  development  of  the  two  IT 
applications  and  if  so,  if  one  will  effectively  meet  requirements.  Additionally,  the  issues 
involved  in  producing  the  needed  infonnation  need  to  be  identified.  In  an  attempt  to 
address  these  requirements,  this  research  focuses  on  four  primary  questions: 

1 .  What  are  the  customer  needs? 

2.  What  does  each,  WST  and  AMST,  provide? 

3.  How  well  does  each  application  meet  the  needs? 

4.  How  supportable  is  each  application  in  terms  of  information  needs  versus  existing 
systems? 
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Limitations 


The  IT  applications  that  are  compared  in  this  study  are  new  to  the  munitions 
community.  As  such,  not  a  lot  of  literature  exists  about  the  applications.  As  mentioned 
earlier,  both  applications  are  web-based,  which  complicates  the  process  of  obtaining  an 
accurate  count  of  actual  users  of  the  system.  Additionally,  WST  is  only  in  use  on  the 
Secure  Internet  Protocol  Router  Network  at  AAC,  Eglin  AFB  (Greg  Jenkins,  May,  2002), 
which  complicates  the  situation  more.  The  users  manual  for  WST  is  still  in  production 
while  the  users  guide  for  AMST  is  only  a  year  old.  Subject  matter  experts  must  be  used 
almost  exclusively  to  facilitate  an  accurate  understanding  of  the  developmental  and 
operational  aspects  of  both  systems.  Even  though  this  is  the  best  time  to  conduct  a  test, 
this  aspect  of  the  study  presents  the  opportunity  for  system  developer  and  potential  user 
bias  toward  the  application  that  may  be  available  and  in  use  in  their  area. 

An  unbiased  scenario  for  the  operational  testing  of  both  applications  must  be 
created  as  well  as  an  adequate  understanding  of  the  operational  aspects  of  the 
applications.  Resistance  to  change  and  a  learning  curve  is  inevitable  when  new 
processes,  ideas,  or  applications  are  introduce  in  an  organization.  To  overcome  these 
aspects  of  change,  this  thesis  effort  focuses  on  the  ease  of  using  the  applications,  the 
actual  performance  of  the  applications,  and  if  the  applications  meet  the  needs  of  the  user. 

Scope 


To  answer  the  research  questions,  the  research  design  will  follow  a  mixed 
qualitative  and  quantitative  paradigm  to  conduct  a  gap  analysis.  Archival  data  and  survey 
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information  will  be  gathered  from  briefings,  documents,  and  interviews  with 
developmental  contractors  for  WST  and  AMST,  AFRL,  and  munitions  personnel  to 
determine  the  capabilities  of  each  system.  Survey  infonnation  will  be  used  to  identify  the 
level  of  difficulty  involved  in  manipulating  the  applications,  the  performance  of  the 
application  and  the  extent  to  which  the  application  meets  user  needs.  Using  the  gathered 
information,  the  gap  analysis  aims  to  identify  differences  between  the  two  systems  as 
well  as  the  relative  merits  of  each. 

Summary 

This  chapter  has  presented  the  overall  context  this  research.  There  is  a  need  for 
an  IT  application  that  can  pull  munitions  data  from  existing  ammunition  information 
systems  and  combine  it  to  provide  a  complete  picture  of  munitions  status  and  capabilities 
for  the  user.  This  research  will  investigate  the  potential  abilities  of  two  proposed 
information  systems  (WST  and  AMST)  to  satisfy  these  IT  needs. 

The  remaining  chapters  of  this  thesis  will  present  a  literature  review,  the  study 
methodology,  findings  and  analysis,  and  a  conclusion  with  recommendations.  The 
literature  review  builds  a  background  as  to  the  importance  and  need  for  a  munitions  IT 
application.  Chapter  three,  the  methodology  chapter,  describes  how  the  research  data 
was  collected  and  analyzed  through  archival  sources  and  survey  data.  Chapter  four 
presents  findings  and  the  analysis.  Chapter  five  examines  the  results  and  presents 
recommendations  for  further  study. 
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II.  Literature  Review 


Expeditionary  Aerospace  Force  and  the  Air  Force  Core  Competencies 

Joint  Vision  2020  outlined  a  new  role  for  the  logistics  community.  “The 
overarching  focus  of  this  vision  is  full  spectrum  dominance  -  achieved  through  the 
interdependent  application  of  dominant  maneuver,  precision  engagement,  focused 
logistics,  and  full  dimensional  protection”  (DoD  JV  2020,  2000).  An  aspect  that  assists 
in  achieving  such  dominance  is  superior  information.  Infonnation  superiority  is  defined 
as  “the  capability  to  collect,  process,  and  disseminate  an  uninterrupted  flow  of 
information  while  exploiting  or  denying  an  adversary’s  ability  to  so  the  same”  (DoD  JV 
2020,  2000).  Accurate  and  timely  logistics  information  plays  such  a  key  role  in  obtaining 
information  superiority  that  it  is  one  of  the  key  tenets  of  Focused  Logistics.  Focused 
Logistics  “will  be  made  possible  through  a  real-time,  web-based  information  system 
providing  total  asset  visibility  as  part  of  a  common  relevant  operational  picture 
effectively  linking  the  operator  and  logistician  across  services  and  support  agencies” 
(DoD  JV  2020,  2000). 

The  USAF  implemented  the  Joint  Chiefs’  vision  for  the  future  and  developed  the 
concept  of  Global  Engagement,  which  required  the  Air  Force  to  organize  under  the 
Expeditionary  Aerospace  Force  (EAF)  concept.  The  EAF  concept  was  a  fundamental 
change  for  the  Air  Force  that  is  grounded  in  responsiveness,  flexibility,  and  combat 
effectiveness.  It  motivated  an  organizational  culture  shift  towards  an  expeditionary 
warrior  mindset  and  provided  a  new  vision  for  how  the  Air  Force  organized,  trained, 


equipped,  and  sustained  aerospace  forces  to  meet  the  requirements  of  national  military 
strategy  and  the  challenges  of  the  changing  global  security  environment  (Brian  Peters 
Presentation,  1999).  In  order  to  achieve  the  goals  of  the  EAF  concept,  the  U.S.  Air  Force 
organized  into  Aerospace  Expeditionary  Forces  or  AEFs.  These  AEFs  are  airpower 
packages  tailored  to  quickly  deploy  and  meet  the  tasking  of  the  national  command 
authorities  and  the  regional  commander  in  chief.  Each  AEF  has  equivalent  capabilities 
made  up  of  fighter  squadrons,  bomber  squadrons,  theater  lift,  tanker  forces,  and 
appropriate  combat  support  units.  The  packages  bring  all  the  necessary  capabilities  that 
are  critical  for  combat  success  to  the  table  including  “the  ability  to  achieve  air 
superiority;  suppression  of  enemy  air  defenses;  air-to-ground  bombing  and  precision 
munitions;  mobility;  combat  support;  and  tactical  level  leadership”  (HQ  USAF  website). 
However,  the  packages  are  made  up  of  units  from  across  the  Air  Force,  which  makes  it 
essential  to  have  comprehensible  and  timely  infonnation  to  support  mission 
requirements.  This  research  endeavor  focuses  on  the  availability  of  timely  and 
dependable  munitions  information. 

The  previously  mentioned  capabilities  are  met  through  the  Air  Force’s  core 
competencies.  “Core  competencies  are  at  the  heart  of  the  Air  Force’s  strategic 
perspective  and  thereby  at  the  heart  of  the  Service’s  contribution  to  our  nation’s  total 
military  capabilities”  (AFDD  1,  1997).  The  Air  Force’s  core  competencies  are  Air  and 
Space  Superiority,  Precision  Engagement,  Information  Superiority,  Global  Attack,  Rapid 
Global  Mobility,  and  Agile  Combat  Support.  Each  of  these  is  dependent  on  timely  and 
accurate  munitions  infonnation.  In  addition  to  the  core  competencies,  Air  Force  Doctrine 
Document  2-4  identifies  five  core  combat  support  principles:  responsiveness, 
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survivability,  sustainability,  time-definite  re-supply,  and  information  integration  (1999). 
The  availability  of  timely  munitions  information  is  vital  in  providing  the  flexibility  to: 
support  responsiveness  for  munitions  needs;  meet  the  requirements  for  time-definite  re¬ 
supply  of  munitions  to  aircraft  and  areas  of  conflict;  sustain  deployed  forces  when 
needed;  and  to  play  a  part  in  integrating  information  to  improve  command  and  control 
(C2). 


Command  and  Control 


The  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms  defines 
command  and  control  as: 


The  exercise  of  authority  and  direction  by  a  properly  designated 
commander  over  assigned  and  attached  forces  in  the  accomplishment  of 
the  mission.  Command  and  control  functions  are  performed  through  an 
arrangement  of  personnel,  equipment,  communications,  facilities,  and 
procedures  employed  by  a  commander  in  planning,  directing, 
coordinating,  and  controlling  forces  and  operations  in  the  accomplishment 
of  the  mission.  (JP1-02,  2000) 

The  Joint  Chiefs  state  that  there  are  two  main  parts  in  implementing  proper  command  and 
control:  “command  structures  and  processes,  and  the  infonnation  systems  and 
technologies  that  are  best  suited  to  support  them”  (DoD  JV  2020,  2000). 

Air  Force  doctrine  identifies  two  tenets  of  C2.  The  first,  unity  of  command,  is  an 
essential  aspect,  which  requires  all  parties  to  understand  the  chain  of  command  and 
adhere  to  it.  “Vertical  information  flow  is  fundamental  to  centralized  control”  (AFDD  2- 
8,  2001).  This  flow  of  information  provides  commanders  with  the  information  they  need 
to  make  good  decisions.  “Horizontal  information  flow  is  essential  for  common 
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situational  awareness”  and  enhances  operator  initiative  and  reduces  uncertainty  between 
peer  levels  (AFDD  2-8,  2001).  Centralized  control  and  decentralized  execution  reinforces 
the  tenet  of  unity  of  command. 

The  second  tenet  is  informed  decision  making.  “Command  and  control  should 
support  an  informed  and  timely  decision-making  process  at  all  levels  of  command” 
(AFDD  2-8,  2001).  Improving  the  timeliness  and  accuracy  of  munitions  information 
directly  supports  this  tenet.  When  the  Joint  Chiefs  published  JV  2020,  they  continued 
explaining  that  the  tenet  includes  “planning,  directing,  coordinating,  and  controlling 
forces  and  operations,  and  is  focused  on  the  effective  execution  of  the  operational  plan,” 
but  emphasized,  “the  central  function  is  decision  making”  (DoD  JV  2020,  2000). 
Command  and  control  activities  can  benefit  from  improvements  in  information 
centralization  and  accessibility.  By  making  information  readily  available,  decision 
makers  can  come  to  informed  decisions  in  a  more  time  efficient  manner. 

Decision  Support  Systems 

A  Decision  Support  System  (DSS)  is  used  to  aid  decision  makers  in  their  efforts 
to  acquire,  organize,  process,  present,  and  use  information  for  making  decisions. 
Characteristics  of  a  good  DSS  focuses  on  “increased  individual  and  organizational 
effectiveness  rather  than  on  increased  efficiency  in  processing  masses  of  data”  (Alter, 
1980).  Vlatko  Ceric  (Ceric,  1997)  and  Sprague  (Sprague,  1989)  combined  the  work  of 
Alter  (Alter,  1980)  and  Keen  (Keen,  1981)  with  their  own  work  to  outline  six  necessary 
qualifications  for  good  DSSs.  Ceric  summarized  DSS  characteristics  as  being  able  to: 
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1 .  Assist  the  users  in  semi-structured  decision  tasks 

2.  Support  managerial  judgment 

3.  Improve  the  effectiveness  of  decision-making 

4.  Be  used  by  non-computer  specialists  in  an  interactive  manner 

5.  Combine  use  of  models  with  data 

6.  Adapt  to  the  decision-making  approach  of  the  user  (Ceric,  1997). 

The  above  themes  are  widely  agreed  upon  standards  for  DSSs.  Sprague  theorized  that 
because  improving  performance  is  the  ultimate  objective  and  knowledge  workers  are  the 
clientele,  any  DSS  framework  must  be  created  adjusting  for  variation  (Sprague,  1989). 
The  DSS  does  not  have  to  accommodate  all  levels  of  management  but  it  should  consider 
the  relationship  the  user  has  with  system.  Herbert  Simon  first  identified  this  concept  as 
“bounded  rationality”  (Simon,  1945).  Simply  put,  any  DSS  framework  must  account  for 
the  limiting  capabilities  of  the  user. 

Supporting  munitions  operations  is  not  an  easy  task.  However,  highly  capable 
and  inexpensive  computer  systems  are  readily  available  that  can  assist  in  the  munitions 
decision-making  process  when  the  appropriate  software  is  installed  to  provide  the 
necessary  intelligence. 


The  situation  is  improved  when  decision  support  system  designers 
incorporate  interface  considerations  so  that  (a)  the  complexity  of  the 
domain  is  appropriately  captured,  and  (b)  this  representation  is  effectively 
communicated  to  the  user.  A  user-friendly  DSS  should  support  a  variety 
of  user  behavioral  characteristics  and  needs...  A  DSS  should  consist  of  a 
human-computer  interface  that  accurately  and  simply  conveys  the 
information  in  the  computer  to  the  user,  and  the  user’s  desires  to  the 
computer.  (Smith,  1998) 

Due  to  the  number  of  actions  required  to  get  good  information,  munitions 
personnel  often  keep  paper  logs  or  local  spreadsheets  and/or  databases  in  addition  to  the 
official  entries  in  munitions  legacy  systems.  This  is  especially  true  for  data  that  is  time 
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sensitive  or  needed  on  a  recurring  basis.  Although  these  actions  solve  problems,  they 
also  result  in  loss  of  data  integrity  and  neglect  of  feedback  and  updates  to  the  contributing 
legacy  systems.  When  copies  of  data  are  kept  locally  as  well  as  in  legacy  systems, 
differences  occur  and  the  question  arises  as  to  which  source  of  information  is  correct. 

The  challenge  is  to  provide  munitions  personnel  with  the  convenience,  usefulness,  and 
timeliness  of  local  copies  of  combined  legacy  system  data,  while  preserving  the  data 
management  of  the  source  data  provided  by  the  legacy  systems  (Paperless  Acquisition 
Initiative,  1997).  The  necessity  for  a  DSS  has  become  increasingly  important  to 
munitions  decision  makers  due  to  the  increasing  complexity  and  rapid  nature  of  the 
decisions  they  must  make.  Even  the  Air  Force  policy  on  centralized  C2  and 
decentralized  execution  expands  the  expected  use  of  a  DSS  to  a  potentially  non- 
managerial  level,  which  translates  into  many  of  the  decisions  responsibilities  being  made 
at  the  job  site. 

A  potential  solution  lies  in  the  use  of  intelligent  agents  and  infonnation  brokers  to 
gather  munitions  information  from  legacy  system  data.  “Intelligent  Agents”  are  defined 
as  computer  programs  that  have  special  skills  that  engage  and  help  humans  in  complex 
tasks  (Paperless  Acquisition  Initiative,  1997).  Intelligent  agents  used  in  conjunction  with 
information  brokers  could  potentially  gather  munitions  data  on  demand,  at  a 
predetennined  time,  or  when  certain  conditions  are  met.  An  intelligent  agent  strives  to 
meet  five  characteristics: 

1 .  Integrated,  supporting  an  understandable,  consistent  interface. 

2.  Expressive,  accepting  requests  in  different  manners  and  displaying 
more  than  characters. 

3.  Goal-oriented,  determining  when  and  how  to  achieve  a  user’s  goal. 
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4.  Cooperative,  collaborating  with  the  user. 

5.  Adaptable  to  different  users  (Paperless  Acquisition  Initiative,  1997). 

A  true  intelligent  agent  should  be  capable  of  autonomous  goal-oriented  behavior,  acting 
as  a  personal  assistant  or  representative  of  the  user  (Paperless  Acquisition  Initiative, 
1997). 

An  example  of  a  simple  intelligent  agent  is  the  POINT  CAST  system  that  scans 
the  world  wide  web  (WWW)  search  engines  directories  for  news  or  special  interest  items 
that  the  user  designates  in  a  user  profile  and  continuously  downloads  the  results  to  the 
users  local  web  browser  (Steve  Powers,  June  26,  2002).  More  sophisticated  intelligent 
agents  mine  for  information  from  data  warehouses.  These  tools  use  several  techniques  to 
extract  and  display  data  and  data  trends  to  users.  These  techniques  include:  rule-based, 
decision  trees,  neural  net-based,  statistical  analysis,  visual  and  fuzzy  logic  discovery, 
knowledge-based,  or  combinations  of  the  techniques  (Scott  Fagan,  June  26,  2002). 

Information  brokers  act  as  intennediaries  between  two  or  more  sources  of  data 
(Clark  Moskop,  June  26,  2002).  They  can  handle  multiple  formats  of  data  from  multiple 
sources  and  help  overcome  the  need  for  expensive  direct  interfacing  of  legacy  systems. 
An  example  of  an  information  broker  given  during  the  LOCIS  Critical  Design  Review 
was  the  Electronic  Data  Intcrchangc/Valuc  Added  Network  ventures  that  supply  common 
interfaces  for  commercial  purchasing  and  supply  systems  (Steve  Powers,  June  26,  2002). 
Broker  examples  today  can  range  from  activities  as  simple  as  finding  infonnation  via 
“spiders”  on  the  WWW  to  more  complex  activities  such  as  building  data  warehouses. 
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Munitions  Databases  and  Decision  Support  Systems 


Computer-assisted  decision-making  is  needed  to  meet  the  increasing  demands  that 
are  put  on  munitions  decision  makers.  A  vast  amount  of  munitions  infonnation  exists  on 
many  different  systems  to  facilitate  informed  munitions  decision-making,  however,  the 
information  is  often  viewed  in  absence  of  each  other.  Munitions,  missiles,  and  systems 
are  tracked  throughout  the  munitions  process  using  various  information  systems  and 
“homegrown”  programs.  CAS  is  the  primary  munitions  reporting  system  used  by  the  Air 
Force.  CAS-Base  (B)  is  used  at  the  base  level  to  provide  management  support  and 
custody  control.  CAS-Deployable  (D)  is  a  deployable  version  of  CAS-B.  Although  it  is 
not  as  functionally  complex  as  CAS-B,  CAS-D  is  able  to  maintain  current  and  accurate 
munitions  information.  CAS-Command  (C)  provides  Major  Commands  (MAJCOMs) 
command-level  munitions  visibility  and  planning  functions.  CAS-A,  the  Ammunition 
Control  Point  (ACP),  provides  national-level  munitions  planning  and  control  for  the  ACP 
at  the  Air  Logistics  Centers  and  Air  Staff.  CAS-B  provides  asset  and  expenditure 
information  to  CAS-C  and  CAS-A.  Requisitions  are  also  passed  from  CAS-B  to  CAS-C, 
which  feeds  it  to  CAS-A.  Munitions  allocations  are  passed  from  CAS-C  to  CAS-B  so 
that  the  system  can  provide  conventional  and  nuclear  munitions  maintenance  and 
inventory  control  functions  such  as:  requirements  forecasting,  storage  planning, 
movement  execution,  complete  round  assembly,  resource  management,  production 
scheduling,  vertical  reporting,  support  flow  plans,  inventory  control,  and  stockpile 
conditions  tracking.  CAS-D  supports  deployed  units  and  the  Regional  Ammunition 
Control  Point  with  munitions  information.  Munitions  asset  data  from  CAS-B  are  loaded 
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into  CAS-D  by  disk  or  manually  entered  from  a  hard  copy.  Likewise  data  from  CAS-D 
are  retuned  to  CAS-B  by  disk  or  hard  copy. 

D023K  and  D035K  are  depot-level  information  systems  used  in  the  munitions 
community.  D023K  manages  shipments;  receipts;  asset  movements  from  one  location  to 
another;  inspections,  test  and  warranty  infonnation;  as  well  as  inventory  counts.  It  also 
reports  the  status  of  requisitions  and  munitions  available.  D035K  is  the  accountable 
wholesale  and  retail  receipt  and  shipment  system.  D035K  maintains  the  accountable 
record  of  the  munitions  asset  inventory  and  feeds  asset  balances  to  CAS-A.  Asset  data 
are  fed  from  both  systems  to  CAS-A  and  on  to  the  Global  Transportation  Network 
(GTN).  The  GTN  system  then  tracks  in  transit  munitions  through  the  shipment  and 
receipt  process. 

Other  infonnation  systems  used  to  assist  in  munitions  operations  include:  the 
Theater  Allocation  Buy-Budget  System  (TABBS),  which  is  used  to  determine  allocations 
and  Buy  Budget  based  on  requirements.  The  data  are  fed  to  CAS-A.  Allocations  and 
Requirements  are  also  fed  from  TABBS  to  the  Ammunition  Control  Point  Analysis  Tool 
(ACPAT).  ACPAT  provides  the  allocations  and  requirements  to  CAS-A  as  well  as 
analysis  reports.  The  Tactical  Missile  Record  System  tracks  configuration  data  and 
maintenance  requirements  for  missiles  while  Munitions  Control  2000  tracks  all  munitions 
maintenance  actions.  The  Core  Automation  Maintenance  System  is  also  used  in 
munitions  to  track  personnel  training  actions  and  requirements. 

To  make  effective  decisions  concerning  munitions  issues,  munitions  personnel 
often  require  data  from  one,  if  not  all,  of  the  legacy  systems  previously  mentioned.  To 
obtain  the  data,  they  access  the  legacy  system  or  systems  that  are  suspected  of  having  the 
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data,  review  large  quantities  of  data  to  find  what  they  need,  call  or  talk  to  other  people, 
request  reports  and  then  either  combine,  print,  copy,  or  move  the  data  to  a  location  where 
it  is  used.  This  process  is  time  consuming  and  usually  requires  a  person  to  have  adequate 
computer  skills.  Whether  all  the  data  is  found  or  not  usually  depends  on  the  skill  and 
attention  to  detail  of  the  person  perfonning  the  search  and  the  amount  of  time  allowed 
performing  the  search.  Due  to  continual  offline  or  inefficient  information  systems, 
spreadsheets  and  work-around  programs  are  often  developed  to  support  munitions 
operations.  Munitions  planners  and  users  need  a  source  for  real-time  munitions 
information  that  can  provide  an  accurate  picture  of  munitions  capabilities  in  a  user- 
friendly  environment  that  avoids  the  “homegrown”  remedies. 

If  operations  continue  as  they  do  today,  munitions  personnel  will  be  expected  to 
know  what  information  is  needed,  where  to  find  it,  and  how  to  access  the  necessary 
systems  to  get  the  information.  They  will  also  need  to  have  the  time  and  talent  to  find, 
retrieve,  and  combine  the  data  to  create  meaningful  munitions  information.  Through  the 
use  of  intelligent  agents  and  information  brokers,  a  DSS  could  be  developed  to 
potentially  enable  munitions  personnel  to  ask  for  information  in  simple  tenns  without 
knowing  exactly  where  the  information  is.  The  benefits  would  be  the  quick  availability 
of  reliable  information  whenever  it  was  needed;  the  elimination  of  separate  records  or 
other  local  documents,  along  with  the  required  time  to  maintain  them;  and  the 
maintainability  of  accurate,  timely  information  in  munitions  legacy  databases. 
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Discussion  on  Warrior  Support  Tool  and  Agile  Munitions  Support  Tool 
The  Warrior  Support  Tool 

No  matter  what  methodology  is  followed  in  the  development  of  a  DSS,  the  system 
must  do  what  it  was  developed  to  do  as  well  as  be  usable  by  the  people  it  was  developed 
for.  The  Warrior  Support  Tool  was  designed  to  track  critical  munitions.  Real-time 
munitions  information  is  updated  from  remote  locations  and  integrated  into  the  analysis 
tool  to  give  weapon  planners  the  ability  to  make  timely  decisions  on  how  to  manage, 
respond  to,  and  re-supply  areas  of  weapons  deficiencies,  to  ensure  that  the  war  fighter  has 
the  weapon  of  choice  and  need  (Rob  Roy,  July  11,  2002). 

Timely  information  about  the  availability  of  combat  ready  munitions  assets  is 
critical  in  the  execution  of  aerospace  combat. 

Two  hundred  years  ago  Napoleon  declared  that  successful  campaigns 
were  based  on  timely  information.  The  Warrior  Support  program  is 
designed  to  identify,  track,  and  furnish  predictive  forecasting  of  munitions 
within  the  USAF  inventory  and  provide  a  real-time  status  update  at  a 
Department  of  Defense  Identification  Code  (DODIC)  level.  (Rob  Roy, 

July  11,2002) 

The  application  is  a  web-based  system  that  consists  of  software,  system  hardware,  and  a 
personal  computer  display,  which  is  structured  on  using  server  connectivity  to  obtain 
munitions  data. 

Through  interviews  with  WST  developers  and  the  DSI  web  page,  the  system 
capabilities  were  identified.  In  the  tracking  mode,  a  list  of  munitions  at  a  selected 
location  can  be  displayed.  The  total  quantity  of  All  Up  Rounds  (AURs)  for  each 
configuration  can  be  calculated  and  displayed  based  on  the  availability  of  DODIC  items 
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required  to  build  up  the  selected  AUR.  The  user  can  select  a  unique  configuration  from 
an  available  list  of  munitions  and  see  a  DODIC  breakdown  for  the  particular 
configuration.  A  listing  of  DODIC  items  by  quantity  that  make  up  the  selected  munitions 
and  the  limiting  DODIC  item(s)  are  also  presented  in  this  mode.  Additionally,  a  picture 
of  the  selected  munitions,  the  DODICs,  and  a  detailed  description  of  the  asset  are 
provided.  The  application  can  also  identify  airbases  that  store  the  needed  components  to 
assist  munitions  planners  in  understanding  the  war-ready  inventory  of  munitions  assets. 

In  the  prediction  mode,  the  information  changes  to  a  screen  allowing  a  priority 
build-up  of  munitions  to  meet  desired  scenarios  or  an  evaluation  of  the  capability  to  meet 
a  desired  selection  and  quantity  of  munitions  from  the  DODICs  available.  The  user  can 
select  munitions  for  specific  targets  and  see  the  availability  of  DODIC  resources  for  the 
selected  munitions  at  selected  locations.  This  infonnation  leads  to  the  capability  of 
performing  an  analysis  of  munitions  availability  versus  munitions  needs. 

In  both  the  tracking  and  prediction  modes,  the  user  is  provided  with  a  network  of 
locations  that  represent  the  critical  supply  points  from  depots  and  air  bases  storing 
munitions  assets.  From  this  display,  the  decision-maker  can  select  among  any  of  the 
locations  to  gain  insight  into  the  availability  of  the  type  and  quantity  of  munitions  at  each 
location.  In  addition,  the  user  can  select  a  specific  type  of  munitions  and  see  a  worldwide 
view  of  the  quantity  of  that  asset  by  location. 

The  WST  application  allows  for  two-way  communication  between  Munitions 
Control  personnel  at  any  location  and  the  database  information  using  nothing  more  than  a 
browser.  This  feature  allows  Munitions  Controllers  to  access,  verify,  and  communicate 
critical  infonnation  to  the  appropriate  personnel  in  a  real-time  format.  Effective 
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implementation  of  WST  may  improve  the  availability  of  infonnation  for  the  war  fighter 
as  well  as  provide  a  capability  for  improved  efficiency  of  Air  Force  munitions  planning 
and  operations  for  the  future. 

The  Agile  Munitions  Support  Tool 

The  Agile  Munitions  Support  Tool  is  also  a  web-enabled  IT  application  that  can 
be  used  by  Air  Force  munitions  managers  to  achieve  a  higher  level  of  munitions 
accountability. 

AMST  automates  a  number  of  programs,  centralizing  and  integrating  the 
data  from  systems  such  as  CAS,  GTN,  and  the  Joint  Hazard  Classification 
System  (JHCS).  AMST  was  developed  in  response  to  the  lessons  learned 
from  U.  S.  military  intervention  in  Kosovo  and  the  Air  Force’s  Munitions 
Business  Process  Reengineering  effort.  (Synergy,  2002) 

The  Air  Staff  identified  a  need  to  consolidate  stockpile,  transportation,  and  hazard  safety 

data  thereby  providing  total  asset  visibility  for  logistics  analysis  and  contingency 

assessment  in  support  of  mission  requirements  (Kevin  Spease,  August  12,  2002).  AMST 

came  into  existence  to  satisfy  this  need. 

The  application  provides  integrated,  up-to-date  asset  visibility  for  munitions 
personnel  at  locations  worldwide  to  support  mission  requirements.  AMST  was  designed 
to  facilitate  easy  interface  with  the  CAS-A  database  (Kevin  Spease,  August  12,  2002).  It 
allows  different  levels  of  munitions  management  to  query  and  extract  asset  balances, 
requisitions,  allocations,  levels,  and  indicative  and  other  ancillary  data  with  a  set  of  easy- 
to-use  analytical  tools.  The  system  is  available  during  war,  as  well  as  in  peacetime,  with 
no  degradation  in  functionality  or  loss  of  data  integrity  (Synergy,  2002). 
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AMST  was  developed  in  the  Oracle  8i  architecture.  The  Munitions  Agile  Combat 
Support  Model  (MACSM)  is  currently  being  developed  as  part  of  the  AMST  tool. 
MACSM  will  enable  munitions  personnel  to  assess  critical  munitions  variables,  perform 
What-If  analyses  and  report  munitions  support  requirements  (Synergy,  2002).  Integrated 
with  the  total  asset  visibility  provided  in  AMST,  this  tool  will  enable  the  Air  Staff  and 
MAJCOMs  to  assess  not  only  combat  munitions  requirements  but  training  assets  as  well. 


Information  System 

Systems  Administrator 


Figure  1.  AMST  Information  Flow  (Adapted  from  AMST  Training  Manual) 

Figure  1  depicts  the  flow  of  data  from  various  sources  into  AMST.  AMST 
receives  feeds  from  CAS-A,  GTN,  JHCS,  and  the  Army  system.  One  must  understand 
that  AMST,  as  well  as  WST,  are  only  as  good  as  the  data  they  receive.  AMST  has  the 
capability  to  maintain  data,  which  helps  munitions  managers  track  assets  that  have  been 
expended,  received  or  shipped  that  CAS-A  has  not  reported  (Synergy,  2002).  The  AUR 
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process  provides  asset  visibility  not  only  to  the  current  stockpile  but  also  the  built-up 
status  of  the  stockpile.  The  Expenditure  Query  Screen  is  set  up  in  a  way  where  you  must 
select  the  date  range  for  the  expenditure  rates  you  are  trying  to  retrieve.  You  can  either 
specify  a  date  range  by  month  and  year  or  select  one  or  more  fiscal  years  (Synergy, 
2002).  The  in  transits  query  screen  compares  open  CAS-A  requisitions  with  GTN  cargo 
records  that  represent  the  set  of  current  in  transit  records.  GTN  Cargo  data  is  also 
available  by  itself  through  this  screen.  The  Requisition  History  mode  shows  all  available 
in  transit  assets  and  requisition  information  on  those  assets.  The  available  information  is 
put  into  the  system  at  different  times  from  various  sources,  forming  a  history  of  a 
requisition.  Another  very  useful  feature,  the  Items  of  Special  Interest  Report  (ISIR) 
allows  authorized  users  to  define  a  set  of  ISIR  reference  numbers  (Synergy,  2002). 

Summary 

The  way  the  United  States  military  plans  to  fight  and  support  the  war  fighter  has 
changed.  However,  very  little  has  changed  in  the  way  we  gather  information  to  improve 
that  support.  A  lot  of  munitions  data  is  collected  and  documented,  but  it  is  scattered 
through  many  disparate  databases  and  is  often  untimely.  A  great  deal  of  time  and  effort 
is  also  spent  manually  pulling,  compiling,  and  analyzing  data  to  provide  meaningful 
information.  There  exists  a  need  for  an  IT  application  that  can  automatically  pull 
munitions  data  from  ammunition  information  systems  and  combine  that  data  to  paint  a 
complete  picture  for  the  user.  The  purpose  of  this  research  is  to  investigate  the  ability  of 
WST  and  AMST  to  meet  this  need.  The  research  will  also  ascertain  how  well  each 
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application  meets  the  decision  support  needs  of  the  retail  munitions  community  as  well  as 
identify  strengths  and  weaknesses  of  WST  versus  AMST.  Additionally,  the  research  will 
attempt  to  predict  the  ease  of  manipulating  each  application. 
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III.  Methodology 


Introduction 

This  chapter  describes  the  methodology  used  in  conducting  the  gap  analysis  and 
administering  the  user  surveys.  A  gap  analysis  is  a  survey  instrument  used  to  determine 
the  gaps  between  a  service  offered  and  a  customer’s  expectations  (Foster,  2001).  The  gap 
analysis  in  this  study  aims  to  identify  differences  between  the  needs  of  the  user  and  the 
abilities  of  WST  and  AMST.  With  the  assistance  of  the  data  collected  from  user  surveys, 
the  analysis  exposed  gaps  that  existed  between  the  needs  and  abilities  of  the  applications. 
Additionally,  pertinent  information  about  the  applications  was  gathered  and  a  Likert  scale 
was  used  to  assist  in  addressing  the  identified  gaps  as  well  as  the  importance  of  the  other 
features  that  are  present  in  the  applications. 

Description  of  The  Gap  Analysis 

The  gap  analysis  process  consisted  of  five  phases.  The  phases  were  developed 
based  on  discussions  in  Managing  Quality:  Gap  Analysis  (Foster,  2001). 

Phase  I  -  Identified  the  munitions  user’s  needs 

Phase  II  -  Identified  application  capabilities 

Phase  III  -  Consisted  of  a  matching  exercise 

Phase  IV  -  Identified  the  recognized  gaps 

Phase  V  -  Provided  recommendations  for  further  action. 

Each  phase  is  described  in  more  detail  in  the  following  pages. 
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Phase  I:  Determine  Munitions  User’s  Needs 

This  phase  of  the  analysis  sought  to  determine  the  needs  of  the  munitions  user. 
To  conduct  this  process,  a  content  analysis  was  conducted  by  gathering  information 
through  literature,  interviews,  and  needs  assessment  documents.  At  the  conclusion  of 
Phase  I,  a  list  of  needs  for  a  munitions  IT  application  was  developed.  To  identify  the 
needs,  a  matrix  as  depicted  in  Figure  2  was  used.  This  matrix  provided  the  baseline  for 
the  gap  analysis. 


Figure  2.  Needs  Matrix  Example 


Phase  II:  Determine  Application  Capabilities 

The  second  phase  of  the  analysis  sought  to  identify  the  capabilities  of  the 
applications  by  conducting  an  analysis  of  WST  and  AMST.  Application  capabilities 
were  identified  using  archival  data  in  the  following  formats: 

1 .  Presentations  from  system  developer 

2.  Future  planning  documents  available  via  the  web 

3.  Subject  matter  expert  interviews 

4.  Literature  devoted  to  WST  and  AMST  application  capabilities 

5.  Manipulation  of  the  applications. 

In  addition,  a  survey  was  developed  for  munitions  users  to  identify  the 
capabilities  and  ease  of  manipulating  the  applications.  Like  the  needs  assessment,  the 
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capabilities  were  organized  in  a  matrix  with  each  column  listing  the  application  and  each 
row  representing  a  capability.  Figure  3  provides  an  example  of  the  capability  matrix. 

The  result  of  Phase  II  provided  a  picture  of  the  capabilities  of  WST  and  AMST. 
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Figure  3.  Capability  Matrix  Example 


Phase  III:  Conduct  Matching  Analysis 

A  matching  analysis  was  conducted  in  two  sub-phases;  an  infonnation  phase  and 
survey  phase,  to  determine  where  needs  and  capabilities  matched  up.  The  infonnation 
phase  helped  in  identifying  user  needs  and  application  capabilities  through  interviews  and 
archival  data.  The  survey  phase  collected  data  to  assist  in  ensuring  the  validity  of  the 
user  needs  that  were  identified  and  the  application  capabilities  that  were  listed.  A  matrix 
displaying  user  needs  along  the  rows  and  system  capabilities  along  the  columns  was 
developed  to  facilitate  the  needs  and  capability  matching  exercise  based  on  infonnation 
obtained  from  the  subject  matter  expert  interviews,  archival  data,  and  user  surveys.  In 
conducting  the  matching  analysis,  any  needs  that  were  not  met  by  a  capability  presented  a 
gap  and  were  labeled  with  a  minus  sign  (-).  Needs  that  were  met  by  an  application 
capability  identified  a  match  and  were  labeled  with  an  equal  sign  (=).  Any  capability 
present  without  an  identified  need  presented  a  capability  in  excess  of  a  need  and  was 
labeled  with  a  plus  sign  (+).  Figure  4  provides  an  example  of  the  matching  matrix.  Once 
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the  matching  analysis  matrix  was  completed,  existing  gaps  were  identified  for  further 
analysis  in  Phase  IV. 
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Figure  4.  Matching  Analysis  Matrix  Example 


Phase  IV:  Gap  Analysis 

Phase  IV  sought  to  take  all  gaps  uncovered  in  Phase  III  and  address  them  based 
upon  their  level  of  importance  to  the  munitions  planner.  One  of  the  goals  of  this  research 
was  to  ascertain  if  one  system  would  suffice  for  the  munitions  planner.  The  final  product 
accompanied  by  inputs  from  the  user  surveys  administered  to  personnel  in  the  munitions 
community,  will  provide  decision-makers  with  a  list  of  strengths  and  weaknesses  for  both 
systems.  The  product  will  also  identify  the  perfonnance  capabilities  and  approval  of  each 
system  by  the  user.  Examples  of  the  questionnaires  that  were  used  are  presented  in 
Appendix  A  of  this  study. 

Phase  V:  Recommendations 

Taking  the  results  of  Phase  IV,  the  study  concluded  with  recommendations  for 
action  and/or  further  research  to  eliminate  the  gaps.  The  analysis  provides  decision 
makers  with  an  accurate  identification  of  the  needs  of  the  users  as  well  as  the  capabilities 
of  the  systems.  It  also  identifies  if  one  system  will  suffice  in  the  munitions  community. 
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Data  Collection 


As  previously  mentioned,  two  sub-phases  were  used  in  collecting  the  data  to 
allow  for  the  matching  analysis  to  take  place.  The  infonnation  phase  used  interviews  and 
archival  data  to  obtain  needed  information  as  to  user  needs  and  application  capabilities. 
The  survey  phase  collected  data  to  assist  in  ensuring  the  validity  of  the  user  needs  that 
were  identified  and  the  application  capabilities  that  were  listed.  The  Phases  are  discussed 
in  more  detail  in  the  following  paragraphs. 

Information  Phase:  Conducting  Interviews 

Subject  matter  expert  interviews  produced  the  majority  of  the  information  used  in 
this  research.  An  interview  is  a  method  for  obtaining  needed  information,  facts,  and 
opinions  held  by  subject  matter  experts.  It  is  described  as  a  “conversation  with  a 
purpose”  (Fowler  and  Mangione,  1990).  A  benefit  of  conducting  interviews  lies  in  the 
fact  that  the  one-to-one  nature  of  the  interview  allows  for  concerns  and  questions  to  be 
directly  addressed  (Leedy,  2001).  Mistakes  and  misunderstandings  are  quickly  identified 
and  cleared  up  in  an  interview  environment. 

The  interview  process  consisted  of  three  steps:  planning  the  interview,  running  the 
interview,  and  reporting  the  results  of  the  interview.  In  following  the  steps: 


Determine  what  infonnation  is  desired  and  prepare  an  interview  schedule. 
The  interview  schedule  should  be  a  set  of  topics  and/or  questions  that  need 
to  be  answered  to  obtain  the  necessary  information.  Once  the  interview 
schedule  is  developed,  decide  on  the  order  in  which  the  topics  will  be 
covered.  Develop  a  prompt  and  an  explanation  of  each  topic  to  ensure  that 
a  tactic  exists  to  assist  in  asking  for  the  information  and  explaining  the 
topic  if  the  interviewee  does  not  understand  the  prompt.  Finally,  decide 
how  to  record  the  responses.  The  avenues  available  in  order  of  preference 
are:  your  memory,  concurrent  written  notes  by  yourself,  tape  recorder,  or 
video  (Preece  et  al,  1994). 
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From  that  point,  the  intended  audience  was  contacted  to  ensure  that  the  topics  developed 
would  facilitate  obtaining  and  presenting  meaningful  and  useful  interview  results. 

The  biggest  danger  in  using  interviews  as  a  method  of  data  gathering  is  the 
unstructured  nature  of  reporting  the  resulting  data,  which  can  easily  be  misinterpreted  or 
censored  (Leedy,  2001).  One  must  remain  unbiased  in  reporting  results  of  interviews. 

The  primary  method  of  analysis  that  helps  guard  against  censoring  information  that  is 
difficult  to  handle  or  unexpected  is  to  break  up  the  text  or  notes  from  each  interviewee 
into  a  set  of  simple  propositions,  using  the  interviewee’s  own  words  as  much  as  possible 
(Preece  et  al,  1994).  These  propositions  can  then  become  the  input  to  a  content  analysis 
activity. 

The  interviewing  process  should  not  be  taken  lightly  because  it  can  be  the  source 
of  very  valuable  first-hand  information.  Other  methods  of  collecting  information  from 
users  include  questionnaires,  observation  of  users,  and  user  participation  in  the  context  of 
user  analysis,  and  focus  groups  or  brainstorming.  The  interview  process  accompanied 
with  a  user  survey  presented  itself  as  the  most  viable  method  of  collecting  infonnation  for 
this  research. 

Survey  Phase:  Conducting  User  Surveys 

The  user  survey  is  a  means  of  finding  out  how  a  particular  software  or  web  site  is 
likely  to  be  used  by  a  specific  set  of  users  and  who  those  users  are  likely  to  be  (Laurie 
Quill,  June  2002).  Because  it  is  possible  to  survey  a  large  number  of  users,  usage  profiles 
from  user  surveys  can  be  relied  upon  if  the  correct  methodology  is  developed.  User 
surveys  can  be  analyzed  statistically,  giving  moderately  hard,  objective  data  (Alkin  et  al, 
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1974).  However,  many  sources  of  bias  exist  and  a  poorly  designed  survey  can  do  more 
hann  than  good. 

There  are  a  number  of  methods  in  existence  for  carrying  out  surveys.  The 
usability  net  identifies  the  following  as  core  survey  processes  that  should  be  followed: 
focus  the  survey,  create  the  survey  instrument,  test  the  survey,  conduct  the  survey, 
analyze  the  survey,  and  present  the  results. 

In  focusing  the  survey,  find  out  what  the  major  decision  points  are  and  areas  of 
uncertainty  in  regard  to  the  usage  of  the  product  and  focus  in  on  those  areas.  Akin  et  al.’s 
book,  Evaluation  and  Decision  Making:  The  Title  VII  Experience,  discusses  analysis  of 
decision  points  and  is  a  good  source  of  information  to  begin  this  process. 

A  number  of  guidelines  exist  on  how  to  develop  questions  and  lay  out  surveys  to 
make  them  “respondent  friendly”.  Reliable  questions  focus  on  simple  things  that  occur 
relatively  infrequently.  An  important  element  in  conducting  a  survey  is  to  develop  the 
concept  of  trust  between  yourself  and  the  respondents  to  assist  in  obtaining  good  data  (Lt 
Col.  Swartz,  September  13,  2002).  Use  open-ended  questions  sparingly,  but  always 
include  an  “Other  (please  specify):”  option  at  the  end  of  a  list  of  choices.  Dillman’s  book 
Mail  and  Internet  Surveys:  The  Tailored  Design  Method  is  an  excellent  guide  to 
developing  surveys. 

It  is  absolutely  essential  to  test  the  survey  before  releasing  it.  A  survey  test  must 
be  conducted  in  conditions  as  close  to  the  real  conditions  as  possible  (Lt  Col.  Swartz, 
September  13,  2002).  A  useful  technique  for  testing  a  survey  is  to  conduct  a  “walk 
through”  of  the  survey  with  a  small  number  of  participants  and  ask  them  what  they  think 
and  understand  about  each  question  as  they  go  through  the  survey.  If  your  participants 
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have  questions,  your  actual  survey  population  may  have  questions.  When  the  sampling 
frame  is  established,  everyone  in  the  frame  must  be  afforded  the  same  opportunity  to 
reply  to  the  survey  questions.  Sampling  theory  is  complex,  and  is  best  left  to  a 
statistician  for  an  in  depth  discussion.  However,  in  providing  a  brief  concept  behind  the 
idea  of  the  theory,  one  has  to  state  how  the  total  population  from  whom  the  information 
will  be  obtained  is  defined,  and  then  how  an  unbiased  sample  from  that  population  will 
be  conducted. 

Identifying  how  the  results  will  be  analyzed  should  be  clear.  Coding  and 
tabulating  the  data  should  be  as  automatic  as  possible,  so  as  to  rule  out  possibilities  for 
random  error  or  bias  to  creep  in  (Fowler  and  Mangione,  1990).  A  spreadsheet  is  a  very 
useful  tool  for  keeping  raw  survey  results  because  it  allows  for  easy  manipulation  and  can 
be  exported  to  a  statistics  package  such  as  SPSS. 

How  to  effectively  present  the  results  of  a  survey  is  always  a  challenge.  There  is 
no  right  or  wrong  way  but  there  are  accepted  techniques.  When  presenting  results, 
always  give  the  headline  news  first,  then  follow  that  up  with  a  detailed  analysis  of  how 
you  got  there,  and  finish  with  a  conclusion  based  on  the  data  (Lt  Col  Swartz,  September 
13,  2002).  Opinions  and  biases  are  naturally  fonned  when  data  is  manipulated.  These 
opinions  are  important  and  should  be  presented  but,  “Do  so  by  carefully  marking  them  as 
your  extrapolations  from  the  data  so  as  not  to  confuse  objective  fact  with  subjective 
opinion”  (usabilitynet.org). 

Subjective  Assessment 

An  abundance  of  surveys  exist  but  the  most  appropriate  survey  for  this  research 
endeavor  was  the  subjective  assessment  survey.  Subjective  assessments  tell  the  evaluator 
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how  the  users  feel  about  the  software  being  tested  (Laurie  Quill,  June  2002).  This  is  a 
distinct  difference  from  a  performance  test,  which  tells  how  efficiently  or  effectively 
users  perform  with  the  software.  The  benefit  of  using  a  subjective  assessment  was  that, 
“In  a  discretionary  use  scenario,  user  satisfaction  is  most  probably  the  largest  single  key 
factor,  which  will  influence  the  user’s  decision  whether  or  not  to  continue  with  the 
software. .  .in  a  mandatory  use  scenario,  poor  satisfaction  leads  to  absenteeism,  fast 
turnover,  and  complaints  from  the  workforce”  (usabilitynet.org).  Subjective  assessments 
also  produce  a  list  of  satisfying  and  unsatisfying  software  features,  which  can  be 
especially  useful  during  an  application’s  development. 

It  is  customary  to  use  a  closed-ended  questionnaire  if  one  is  available.  This 
allows  for  the  gathering  of  quantitative  data,  which  minimizes  the  opportunity  for  the 
results  of  the  activity  to  be  vague  and  open  to  interpretation  (Preece  et  al,  1994).  In  this 
research  endeavor,  once  a  questionnaire  was  identified  for  use,  a  group  of  users  to  fill  out 
the  questionnaire  was  identified.  A  profile  section  was  used  to  get  background  data  on 
each  potential  respondent  (e.g.  computer  experience,  job  level,  frequency  of  use  of  the 
software  being  evaluated).  It  was  important  that  the  respondents  were  not  prompted  as  to 
how  they  should  reply  to  questions.  If  a  respondent  complained  that  a  question  was 
inapplicable  or  wrong,  they  were  told  words  to  the  effect  that  it  was  up  to  them  to  make 
their  own  judgment  about  each  question,  and  that  there  were  no  right  or  wrong  answers. 
However,  they  were  encouraged  not  to  leave  questions  blank. 

Likert  Scales 

Likert  scales  provide  a  method  for  presenting  data  in  an  easily  understood  manner 
(Barbara  Masquelier,  March  2002).  Subjective  questionnaires  usually  have  a  scoring 
scheme  based  on  Likert-style  scaling  techniques.  In  fact,  most  questionnaires  only 
require  that  the  scores  obtained  from  respondents  be  summed  together  to  determine  the 
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survey  score.  The  first  step  in  developing  a  Likert  scale  is  to  define  what  it  is  you  are 
trying  to  measure  (Trochim,  2001).  A  scale  can  have  any  number  of  dimensions  but  most 
only  have  a  few.  A  dimension  is  nothing  more  than  a  number  line  and  if  a  construct  is 
measured,  a  decision  must  be  made  as  to  whether  the  construct  can  be  measured  with  one 
or  more  number  lines.  If  one  number  line  can  measure  a  construct,  it  is  probably  a  one¬ 
dimensional  construct  (Trochim,  2001). 

Once  a  construct  of  measure  is  identified,  create  the  potential  scale  items.  These 
should  be  items  that  can  be  rated  on  a  l-to-5  or  l-to-7  Disagree-Agree  response  scale. 
Sometimes  scale  items  can  be  created  from  understanding  of  the  subject  matter. .  .but 
more  often  than  not,  it's  helpful  to  engage  a  number  of  people  in  creating  the  items 
(Trochim,  2001).  The  next  step  is  to  have  a  group  of  people  rate  the  items  using  a  l-to-5 
rating  scale  where: 

1  =  strongly  unfavorable  to  the  concept 

2  =  somewhat  unfavorable  to  the  concept 

3  =  undecided 

4  =  somewhat  favorable  to  the  concept 

5  =  strongly  favorable  to  the  concept 

The  people  that  rate  the  items  are  not  identifying  what  they  believe;  they  are  judging  how 
favorable  each  item  is  with  respect  to  the  construct  of  interest  (Trochim,  2001). 

Upon  verification  of  the  rating  scale,  compute  the  intercorrelations  between  all 
pairs  of  items,  based  on  the  item  ratings  received.  There  are  several  analyses  that  can  be 
conducted  to  assist  in  deciding  which  items  to  keep  for  the  final  scale: 
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1 .  Throw  out  any  items  that  have  a  low  correlation  with  the  total 
(summed)  score  across  all  items... Statistics  packages  make  it  relatively 
easy  to  compute  this  type  of  Item-Total  correlation.  First,  create  a  new 
variable,  which  is  the  sum  of  all  of  the  individual  items  for  each 
respondent.  Then,  include  this  variable  in  the  correlation  matrix 
computation.  As  for  throwing  out  the  low  rated  correlation  items,  there  is 
no  fixed  rule.  Correlations  have  been  eliminated  that  had  total  scores  less 
that  .6. 

2.  Get  the  average  rating  for  the  top  quarter  of  judges  and  the  bottom 
quarter  for  each  item.  Then,  do  a  t-test  of  the  differences  between  the 
mean  value  for  the  item  for  the  top  and  bottom  quarter  judges.  High  t- 
values  mean  that  there  is  a  greater  difference  between  the  highest  and 
lowest  judges.  Items  with  higher  t-values  are  better  discriminators,  so  you 
want  to  keep  these  items.  In  the  end,  you  will  have  to  use  judgment  as  to 
which  items  are  retained.  You  want  a  relatively  small  number  of  items  on 
your  final  scale  (e.g.,  10-15)  and  you  want  them  to  have  high  Item-Total 
correlations  and  high  discrimination  (e.g.,  high  t-values)  (Trochim,  2001). 

There  are  a  variety  of  response  scales  (l-to-7,  l-to-9,  0-to-4)  that  exist  in  Likert 

scaling.  Odd-numbered  scales  have  a  middle  value  that  is  often  labeled  neutral  or 

undecided,  which  is  very  friendly  to  the  respondent  but  may  not  help  the  researcher.  It  is 

possible  to  use  a  forced-choice  response  scale  with  an  even  number  of  responses  and  no 

middle  neutral  or  undecided  choice  (Leedy,  2001).  In  this  situation,  the  respondent  is 

forced  to  decide  whether  they  lean  more  towards  the  agree  or  disagree  end  of  the  scale  for 

each  item.  The  final  score  for  the  respondent  on  the  scale  is  the  sum  of  their  ratings  for 

all  of  the  items. 


Survey  Development 


The  amount  of  satisfaction  experienced  by  a  user  of  an  application  is  a  critical 
measure  of  any  application’s  success.  There  are  many  possible  ways  to  evaluate  the 
human-computer  interface.  Chin  states  that  there  are  five  different  types  of  dependent 
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measures  for  evaluating  interfaces:  speed,  accuracy,  time  to  leam  a  system,  retention  of 
knowledge,  and  user  acceptance  (1988). 


For  many  tasks,  speed  and  accuracy  are  two  related  perfonnance  measures 
that  affect  a  person’s  attitude  toward  a  system.  The  time  it  takes  to  learn  a 
system  and  the  retention  of  acquired  knowledge  over  time  also  affect  the 
utility  of  a  system.  User  acceptance  of  a  system  (i.e.,  subjective 
satisfaction)  is  also  a  critical  measure  of  a  system’s  success.  Although  a 
system  may  be  evaluated  favorably  on  every  performance  measure,  the 
system  may  not  be  used  very  much  because  of  the  user’s  dissatisfaction 
with  the  system  and  its  interface.  (Chin,  1988). 

The  data  for  the  survey  used  in  this  study  came  from  six  sources;  system 
developer  need  assessment  documents,  application  planning  documents,  subject  matter 
expert  interviews,  literature  devoted  to  WST  and  AMST  capabilities,  manipulation  of  the 
applications,  and  existing  survey  tools.  The  key  to  selecting  the  appropriate  data  was  to 
find  variables  that  were  relevant  to  the  research  and  to  obtain  a  large  enough  sample  to 
represent  the  munitions  community  when  the  survey  was  administered.  To  ensure  a  large 
yet  diverse  sample,  munitions  personnel  attending  7-level  training  as  well  as  those 
attending  the  Munitions  Inspector  Course  at  Sheppard  AFB  were  surveyed.  Munitions 
Instructors  at  the  enlisted  and  officer  schoolhouses  and  Munitions  Officers  attending 
classes  were  also  surveyed.  To  complete  the  sampling  population,  survey  sessions  were 
also  conducted  with  6  Aircraft  Maintenance  Officers  with  at  least  1-year  experience  in 
the  munitions  community.  In  all,  65  participants,  3  civilians,  41  enlisted  ranging  from 
Ainnan  First  Class  to  Master  Sergeant,  and  21  officers  ranging  from  Second  Lieutenant 
to  Captain  were  surveyed.  Their  level  of  computer  experience  varied  from  very  little 
software  experience  to  those  that  were  very  capable  software  programmers.  Everyone 
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had  used  Microsoft  Office  applications  in  one  way  or  another  and  60%  of  those  who 
reported  using  CAS  (n=30),  used  the  system  more  than  4  hours  a  week. 

An  adaptation  of  the  Questionnaire  for  User  Interface  Satisfaction  was  used  to 
measure  the  user’s  subjective  rating  of  WST  and  AMST.  The  original  questionnaire 
consisted  of  80  questions.  Six  questions  were  demographics  questions,  six  were  overall 
reaction  ratings  of  the  system,  thirteen  referred  to  the  screen,  seventeen  referred  to  the 
tenninology  and  system  information,  twenty-one  referred  to  learning  the  system,  and 
seventeen  referred  to  the  capabilities  of  the  system.  Every  question,  except  the 
demographics  questions,  had  a  rating  scale  ascending  from  1  on  the  left  to  9  on  the  right. 
The  scales  were  anchored  at  both  ends  with  conflicting  adjectives  so  that  they  were 
always  positioned  in  a  way  that  the  scale  went  from  a  negative  adjective  on  the  left  to  a 
positive  adjective  on  the  right.  Each  scale  also  had  a  ‘not  applicable’  choice. 

The  original  questionnaire  was  modified  into  two  sections  with  a  total  of  49 
questions  asked.  Section  1  consisted  of  eleven  questions  aimed  at  developing  the 
demographics  of  the  sampling  population.  Participants  were  asked  to  identify  their  sex, 
rank,  duty  title,  AFSC,  years  in  that  AFSC,  their  major  command,  time  spent  with  the 
study  application,  time  spent  on  computers  in  general,  time  spent  on  CAS,  their  perceived 
experience  with  computer  applications,  and  to  identify  the  sort  of  tasks  they  performed  on 
computers.  Section  2  was  a  modified  version  of  the  original  questionnaire  with  the 
difference  being  a  rating  scale  of  1  to  7  and  a  reduction  in  the  number  and  types  of 
questioned  asked.  Thirty-eight  questions  were  distributed  across  eight  constructs;  three 
questions  referred  to  the  overall  reaction  rating  of  the  systems,  three  referred  to  the 
screen,  three  to  the  terminology  and  system  information,  eleven  to  learning  the  system, 
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three  to  help  features  of  the  system,  three  to  speed  of  the  system,  and  five  each  on  the 
importance  and  performance  of  the  features.  Three  versions  of  the  survey  were  also 
developed  to  ensure  that  reliable  ratings  were  received  from  the  participants.  The 
reliability  of  the  questionnaire  was  good.  Cronbach’s  a,  an  estimation  of  reliability  based 
on  the  average  intercorrelation  among  items,  indicated  that  the  reliability  in  the  items 
ranged  from  .78  to  .97,  above  the  suggested  measurement  of  .70  the  limit  suggested  for 
research  designed  to  make  decisions  affecting  groups  (Huck  and  Cormier,  1996). 

A  factor  analysis  was  conducted  to  detennine  mathematical  relationships  in  the 
user  surveys  that  would  predict  the  ease  of  use  and  likeability  of  WST  and  AMST.  Prior 
to  performing  the  analysis,  the  data  was  tested  for  suitability.  A  number  of  tests  were 
conducted  in  order  to  test  for  suitability:  a  correlation  matrix,  Bartlett’s  Test  of 
Sphericity,  the  Kaiser-Meyer-Olkin  Measure  of  Sampling  Adequacy,  and  the  anti-image 
correlation.  Evidence,  which  will  be  explained  in  the  next  chapter,  emerged  from  the 
tests  to  prove  that  significant  patterns  existed. 

Procedure 

It  was  important  to  administer  the  questionnaire  in  an  unbiased  manner.  Bias  can 
be  accounted  for  either  statistically  or  methodologically.  For  this  study,  the  latter  was 
chosen  as  a  means  of  controlling  for  bias.  The  experiment  was  conducted  in  a  computer 
lab  with  groups  of  9  -  12  people.  The  applicable  questionnaire  was  distributed  to  the 
participants  as  they  entered  the  room.  The  participants  were  randomly  assigned  with  the 
only  requirement  being  that  the  same  rank  structure  and  number  of  people  was  adhered  to 
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for  each  survey  session.  If  nine  people  in  the  ranks  of  Sra  to  TSgt  were  in  one  group, 
another  group  with  the  same  characteristics  was  required. 

Equivalent  groups  were  processed  one  behind  the  other  to  allow  a  controlled 
environment  to  present  the  power  point  presentation  that  familiarized  the  participants 
with  the  applications  they  would  manipulate.  The  topics  that  were  covered  in  each 
presentation  were  a  brief  history  on  the  development  of  the  applications  and  information 
on  their  features.  The  same  number  of  slides  was  afforded  to  each  application  with  the 
same  number  of  bullet  statements. 

It  is  believed  that  users  tend  to  favor  the  first  thing  they  see.  To  control  this  user 
bias,  the  presentation  and  manipulation  of  the  applications  were  alternated.  For  the 
inspector  course  students,  A1C  to  MSgt,  half  of  the  subjects  received  a  presentation  and 
manipulation  period  on  WST  first  then  the  other  half  received  it  on  AMST  first.  The  7- 
level  course  students,  which  were  all  SSgt,  received  presentation  of  the  AMST 
application  first.  The  enlisted  instructors,  TSgt  to  MSgt,  received  a  presentation  on  WST 
first.  When  the  officers  were  surveyed,  a  presentation  covering  both  applications  was 
given  up  front  and  the  applications  were  ran  on  alternating  computers  to  ensure 
randomness.  The  survey  period  lasted  an  hour  for  each  group.  Scenarios  were  developed 
that  were  common  to  both  applications.  The  users  were  given  25  minutes  to  manipulate 
each  system  and  answer  the  survey  pertaining  to  that  system.  5  minutes  were  allotted  for 
presenting  the  power  point  presentation  on  each  application. 

Paired  Sample  T-Test 

A  Paired  Sample  T-Test  with  a  .05  significance  level  was  conducted  to  determine 
if  there  was  a  significant  difference  between  the  mean  ratings  received  for  the  WST  and 
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AMST  applications.  The  significance  level  of  5%  is  generally  accepted  as  a  cut-off  point 
for  a  significant  versus  a  non-significant  result  (Devore,  2000).  The  null  hypothesis 
states  that  there  is  no  difference  in  the  ratings  or  that  the  average  of  the  differences 
between  the  paired  ratings  is  zero. 

H  0:  p  1  -  p  2  =  0 

The  alternative  hypothesis  states  that  there  is  a  significant  difference  in  the  mean 
ratings  for  WST  and  AMST.  If  the  calculated  P-value  is  less  than  the  given  significance 
level,  the  conclusion  is  that  the  mean  difference  between  the  paired  ratings  is 
significantly  different  from  0. 

H  i :  p  i  -  p  2  ±  0 

A  two-tailed  P-value  was  appropriately  used  because  the  differences  between  the 
ratings  can  occur  in  both  directions:  either  negative  or  positive,  the  mean  of  one  rating 
can  be  smaller  or  bigger  than  the  other.  Since  this  study  is  designed  to  uncover  a 
difference  in  the  two  applications  in  question,  one  should  be  willing  to  accept  that  there  is 
a  difference  even  when  the  P-value  is  as  large  as  .  10  (Devore,  2000).  To  avoid  the 
mistake  of  interpreting  a  shift  in  P-value  (.047  to  .05 1)  as  a  change  from  significance  to 
non-significance,  the  actual  P-values  are  reported  to  enable  the  reader  to  make  his  or  her 
own  interpretation. 

Summary 

This  chapter  began  with  a  description  of  the  gap  analysis  process  and  continued 
with  a  description  of  the  instrument  design,  sample  population,  and  administration  of  the 
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survey.  In  the  following  chapter,  the  results  from  the  application  of  this  methodology 
will  be  presented.  There,  the  comparative  results  between  the  Warrior  Support  Tool  and 
the  Agile  Munitions  Support  Tool  will  be  analyzed  and  the  ideal  application  presented. 
Suggestions  for  further  research  will  also  be  presented. 
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IV.  Results 


Overview 

This  research  began  with  the  objective  of  identifying  how  well  WST  and  AMST 
met  the  decision  support  needs  of  the  retail  munitions  community  and  to  detennine  if  one 
IT  application  would  suffice  in  the  munitions  community.  The  analysis  and  results 
presented  in  this  chapter  are  an  attempt  to  identify  how  well  the  applications  support  the 
munitions  community  and  determine  if  one  will  suffice.  In  order  to  answer  the 
investigative  questions  and  confirm  how  well  the  applications  supported  the  needs  of  the 
munitions  community,  a  gap  analysis  and  survey  were  accomplished  with  personnel  in 
the  munitions  community  having  at  least  3  months  experience  and  Aircraft  Maintenance 
Officers  with  at  least  one  year  of  munitions  experience. 

What  are  the  customer  needs? 

A  content  analysis  was  conducted  and  over  100  different  needs  were  identified 
from  the  needs  assessment  documents,  which  were  eventually  categorized  under  6  major 
headings.  At  the  conclusion  of  the  analysis,  the  list  of  identified  needs  for  a  munitions 
information  technology  application  were  placed  in  rows,  and  the  user  that  identified  the 
need  was  placed  in  the  column  of  the  matrix.  The  needs  matrix  depicted  in  Figure  5  is 
the  result  of  the  content  analysis  and  provides  the  baseline  for  the  gap  analysis. 
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Figure  5.  Needs  Matrix 

The  identified  needs  are  critical  in  the  execution  of  aerospace  combat.  They  all 
help  in  tracking  critical  munitions  to  provide  weapon  planners  the  ability  to  make  timely 
decisions  on  how  to  manage,  respond  to,  and  re-supply  areas  of  weapons  deficiencies,  to 
ensure  that  the  weapon  of  choice  and  need  is  available.  By  tracking,  identifying,  and 
providing  requisition  history  on  assets  and  expenditures,  the  total  quantity  for  a  given 
configuration  can  be  calculated  and  displayed  based  on  the  availability  of  the  DODIC 
items  required  to  build  the  asset.  The  information  also  enables  munitions  planners  to 
identify  airbases  that  store  needed  components,  which  acts  as  a  force  multiplier  by 
furthering  the  planners  understanding  of  the  war-ready  inventory  of  munitions  assets. 


What  does  each  application  provide? 

The  next  step  was  to  develop  the  capabilities  matrix.  Like  the  needs  assessment, 
the  capabilities  were  organized  in  a  matrix  with  each  column  listing  the  application  and 
each  row  representing  a  capability.  The  application  capabilities  were  identified  through 
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the  use  of  presentations  and  interviews  with  system  developers,  planning  documents 
available  via  the  web,  literature  devoted  to  WST  and  AMST  application  capabilities,  and 
manipulation  of  the  applications.  In  addition,  a  survey  was  developed  for  munitions 
personnel  to  identify  the  capabilities  and  ease  of  manipulating  the  applications.  Figure  6 
shows  the  results  of  the  capability  matrix  exercise. 
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Figure  6.  Capability  Matrix 


To  detennine  where  user  needs  and  application  capabilities  were  met,  two 
additional  matrices  were  created.  These  matrices  displayed  user  needs  along  the  rows 
and  the  application  capabilities  along  the  columns.  At  that  point,  a  needs  and  capability 
matching  exercise  was  conducted  using  subject  matter  expert  interviews,  archival  data, 
and  user  surveys  to  ensure  validity.  All  needs  and  capabilities  were  labeled  with  a  minus 
sign  (-)  if  a  gap  existed,  an  equal  sign  (=)  if  a  match  existed,  and  a  plus  sign  (+)  for  a 
capability  in  excess. 
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Figure  7  and  8  provide  the  matching  matrices  for  WST  and  AMST  respectively. 
Once  the  matching  analysis  matrices  were  complete,  gaps  were  identified  to  allow  for 
further  analysis  and  discussion. 
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Figure  7.  WST  Matching  Analysis  Matrix 
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Figure  8.  AMST  Matching  Analysis  Matrix 
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How  well  does  each  application  meet  the  needs? 


The  WST  gap  analysis  identified  the  ability  to  track  in  transit  assets  and  provide 
requisition  history  as  gaps  in  the  application.  During  the  survey  exercise,  munitions 
personnel  identified  the  asset  condition  code  feature  as  the  most  important  feature  in  the 
WST  application  (mean=5.43).  Asset  expenditure  rates  and  balances  were  the  next  two 
most  important  features  respectively  (mean=5.28,  5.28).  To  make  accurate  munitions 
decisions,  planners  must  understand  the  status  of  munitions  inventory,  what  weapons  are 
being  produced,  how  expenditures  may  be  impacting  the  mission,  and  the  status  of  re¬ 
supply  assets.  The  ability  to  track  in  transit  assets  plays  a  major  role  in  planning  future 
munitions  operations  and  production  forecasts.  WST  received  praise  even  though  the  in 
transit-tracking  feature  was  lacking.  As  for  the  requisition  history  feature,  the  application 
did  not  provide  an  in-depth  report  of  requisition  history.  Requisition  history  information 
can  be  obtained  from  the  application  but  it  has  to  be  done  so  through  reverse  engineering 
and  deductive  reasoning,  which  may  lead  to  miscalculations.  Additionally,  extreme 
concern  was  expressed  as  to  the  level  of  security  available  for  the  application.  Munitions 
personnel  acknowledged  the  understanding  that  CAS  was  being  declassified  but  were 
concerned  with  the  fact  that  the  available  munitions  information  was  still  sensitive  in 
nature. 

The  AMST  gap  analysis  did  not  identify  gaps  between  user  needs  and  application 
capabilities.  AMST  received  excellent  praise  but  concern  was  expressed  as  to  the 
security  features  of  the  application,  its  speed,  and  the  fact  that  too  much  information  had 
to  be  manually  input  into  the  application.  The  application  tended  to  bog  down  as 
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searches  were  performed.  Munitions  users  also  favored  the  point  and  click  feature  of 
WST  over  the  required  information  input  feature  of  AMST. 

Based  on  the  results  of  Phase  IV,  recommendations  for  action  and/or  further 
research  were  given  to  complete  Phase  V  of  the  gap  analysis  process.  The 
recommendations  that  were  formulated  will  be  presented  in  the  following  chapter. 

Factor  Analysis 

An  exploratory  factor  analysis  was  accomplished  to  determine  if  one  factor  to 
represent  each  characteristic  would  emerge.  However,  prior  to  performing  the  analysis, 
the  data  had  to  be  tested  to  detennine  if  it  would  be  suitable  for  factor  analysis.  A 
correlation  matrix  of  the  items  was  analyzed  to  determine  a  factor  pattern  in  the  WST 
survey  (see  Appendix  B).  Eleven  item  correlations  on  the  WST  survey  were  .023,  .034, 
.073,  .101,  .119,  .140,  .153,  .199,  .203,  .259,  and  .289,  which  provided  evidence  that 
more  than  one  factor  would  emerge  because  Hair  states  that,  for  factor  analysis, 
correlations  above  .30  give  evidence  that  only  one  factor  is  present  (1995).  The  results  of 
the  correlation  matrix  led  to  an  exercise  in  determining  the  correct  number  of  factors 
present  in  the  survey.  Upon  further  manipulation  of  the  data  in  SPSS,  eight  factor 
patterns  emerged  for  the  survey.  The  factors  were  entitled  overall  satisfaction  of  the 
system,  the  screen  layout,  text  communication  and  terminology  of  the  system,  ease  of 
learning  and  using  the  system,  help  features  of  the  system,  speed  and  functionality  of  the 
system,  importance  of  system  features,  and  perfonnance  of  system  features  respectively. 
All  the  item  correlations  were  above  .549  to  provide  the  necessary  evidence  that  the 
factor  patterns  were  identified. 
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Next,  the  Bartlett’s  Test  of  Sphericity  yielded  chi-squares  of  106.494,  75.284, 
54.777,  660.102,  75.629,  97.441,  351.963,  and  274.589  respectively  with  a  p-value  of 
.000  for  all  results.  The  Bartlett’s  Test  checks  the  null  hypothesis  that  the  variables  in  the 
population  correlation  matrix  are  uncorrelated.  A  p-value  less  than  the  significance  level 
that  is  set,  rejects  the  null  hypothesis  and  gives  evidence  that  the  population  is  correlated. 
Another  test,  the  Kaiser-Meyer-Olkin  Measure  (KMO)  of  sample  Adequacy,  yielded 
.727,  .708,  .722,  .898,  .721,  .726,  .886,  and  .864.  Hair  suggests  values  above  .70  are 
desirable  and  below  .50  are  undesirable  (1995).  Finally,  the  anti-image  correlation 
produced  small  and  negative  numbers,  also  deemed  appropriate  by  Hair  (1995). 
Therefore,  these  test  provided  evidence  that  the  data  was  suitable  for  factor  analysis. 

A  principle  components  analysis  was  also  accomplished  where  factors  with 
eigenvalues  greater  than  1  were  extracted.  The  eigenvalue  is  an  index  that  reflects  how 
much  variance  in  a  survey  item  can  be  accounted  for  each  factor.  The  eight  factors  that 
emerged  had  eigenvalues  greater  than  2.242  with  the  highest  being  8.328.  All  of  the 
factors  explained  over  74.734%  of  the  variance  in  each  factor.  This  data  supported  the 
determination  that  the  developed  scale  was  appropriate  for  measuring  the  ease  of  use  and 
likeability  of  WST.  See  the  factor  loading  in  Table  1. 
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Table  1.  WST  Survey  Construct  Validity 


Construct 

Standardized  a 

KMO 

Chi2 

P-value 

Eigenvalue 

Overall  Satisfaction  WQ1-3 

.8984 

.727 

106.494 

.000 

2.495 

Screen  Layout  WQ4-6 

.8465 

.708 

75.284 

.000 

2.297 

Text  Communication  &  Terminology  WQ7-9 

.8309 

.722 

54.777 

.000 

2.242 

Ease  of  Use/Learning  WQ10-20 

-  Learning  (10-12) 

-  Intuitive  (13-15) 

-Presentation  (16-18) 

-  Novice  Friendly  (19-20) 

.9677 

.898 

660.102 

.000 

8.328 

Online  Help  WQ21-23 

.8873 

.721 

75.629 

.000 

2.450 

Speed  WQ24-26 

.8866 

.726 

97.441 

.000 

2.446 

Importance  of  Features  WQ27-31 

.9716 

.886 

351.963 

.000 

4.491 

Performance  of  Features  WQ32-36 

.9606 

.864 

274.589 

.000 

4.321 

The  same  procedures  were  followed  to  determine  if  a  recognized  factor  pattern 
developed  in  the  AMST  survey  (see  Appendix  C).  However,  only  three  item  correlations 
emerged,  .252,  .255,  and  .283,  which  still  provided  evidence  that  more  than  one  factor 
would  emerge.  Eight  factor  patterns  were  developed  and  the  same  titles  were  given  to  the 
factors.  The  item  correlations  on  the  AMST  survey  were  above  .562,  which  again 
provided  the  necessary  evidence  that  factor  patterns  were  identified.  The  Bartlett’s  Test 
of  Sphericity  yielded  chi-square  values  of  54.864,  100.186,  68.835,  628.779,  99.868, 
164.650,  220.543,  and  247.348  with  p-values  of  .000  for  all  the  results.  The  Kaiser- 
Meyer-Olkin  Measure  of  sample  Adequacy,  returned  values  of  .700,  .738,  .737,  .932, 

.733,  .749,  .894,  and  .886.  The  last  test,  the  anti-image  correlation,  also  produced  small 
and  negative  numbers  leading  to  the  conclusion  that  factor  analysis  could  be  conducted 
on  the  AMST  data.  AMST  principle  components  analysis  was  also  accomplished  with 
eigenvalue  factors  greater  than  1  being  extracted.  The  eight  factors  that  emerged  had 
values  greater  than  2.256  with  the  highest  being  8.305.  The  factors  also  explained  over 
75.207%  of  the  variance  for  each  factor,  supporting  the  determination  that  the  scale  was 
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appropriate  for  measuring  the  ease  of  use  and  likeability  of  the  Agile  Munitions  Support 
Tool.  Table  2  gives  the  factor  loading. 


Table  2.  AMST  Survey  Construct  Validity 


Construct 

Standardized  a 

KMO 

Chi2 

P-value 

Eigenvalue 

Overall  Satisfaction  AQ1-3 

.8343 

.700 

54.864 

.000 

2.256 

Screen  Layout  AQ4-6 

.8896 

.738 

100.186 

.000 

2.458 

Text  Communication  &  Terminology  AQ7-9 

.8622 

.737 

68.835 

.000 

2.352 

Ease  of  Use/Learning  AQ10-20 

-  Learning  (10-12) 

-  Intuitive  (13-15) 

-Presentation  (16-18) 

-  Novice  Friendly  (19-20) 

.9672 

.932 

628.779 

.000 

8.305 

Online  Help  AQ21-23 

.9036 

.733 

99.868 

.000 

2.517 

Speed  AQ24-26 

.9417 

.749 

164.650 

.000 

2.687 

Importance  of  Features  AQ27-31 

.9498 

.894 

220.543 

.000 

4.174 

Performance  of  Features  AQ32-36 

.9655 

.886 

247.348 

.000 

4.394 

Paired  Sample  T-Test 

The  results  of  the  T-test  revealed  that  only  five  survey  questions  (6,  12,  21,  22, 
and  28)  showed  significant  differences  at  a  .05  significance  level.  For  question  number 
6,  the  user  was  asked  about  the  ease  of  going  back  to  a  previous  screen.  The  mean  rating 
for  WST  was  4.90  (SD  =  1.471)  and  the  mean  rating  for  AMST  was  5.34  (SD  =  1.606). 
The  test  was  significant,  t  (57)  =  2.061,  p  <  .05.  Question  12  inquired  about  the  required 
time  to  learn  how  to  use  the  applications.  The  mean  rating  for  WST  was  4.56  (SD  = 

1 .5 12)  and  the  mean  rating  for  AMST  was  4.09  (SD  =  1 .567).  The  test  was  significant,  t 
(54)  =  -2.106,  p  <  .05.  Question  number  21  asked  about  the  quality  of  the  help  messages. 
The  mean  rating  for  WST  was  4.43  (SD  =  1.615)  and  the  mean  rating  for  AMST  was 
4.85  (SD  =  1.660).  The  test  was  significant,  t  (45)  =  2.036,  p  <  .05.  For  question  number 
22,  a  question  about  the  speed  of  the  application  was  posed.  The  mean  rating  for  WST 
was  4.60  (SD  =  1.629)  and  the  mean  rating  for  AMST  was  5.26  (SD  =  1.626).  The  test 
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was  significant,  t  (49)  =  2.557,  p  <  .05.  For  question  number  28,  the  mean  rating  for  WST 
was  4.98  (SD  =  1.375)  and  the  mean  rating  for  AMST  was  4.62  (SD  =  1.423).  The  test 
was  significant,  t  (46)  =  -2.027,  p  <  .05.  However,  a  difference  was  inevitable  for  question 
22  because  two  different  questions  were  asked.  On  the  WST  survey,  the  users  were 
asked  to  rate  the  performance  of  the  asset  location  feature  and  on  the  AMST  survey;  they 
were  asked  to  rate  the  performance  of  the  in  transit  feature.  When  the  confidence  interval 
was  relaxed  to  a  90%  confidence  interval,  only  two  additional  differences  were 
discovered  (questions  24  and  32).  Question  number  24,  asked  users  to  rate  the  time  it 
took  the  applications  to  display  needed  information.  The  mean  rating  for  WST  was  4.94 
(SD  =  1.329)  and  the  mean  rating  for  AMST  was  4.42  (SD  =  1.816).  The  test  was 
significant  at  .10,  t  @1)  =  -1.978,  p  <  .10.  Question  number  32  was  another  paired  set 
where  a  difference  was  inevitable  because  different  questions  were  asked.  For  WST,  the 
users  were  asked  to  rate  the  perfonnance  of  the  condition  code  feature  and  the  mean 
rating  was  5.08  (SD  =  1.499).  For  AMST,  the  users  were  asked  to  rate  the  perfonnance 
of  the  expenditure  feature.  The  mean  rating  for  that  question  was  4.69  (SD  =  1.504). 

The  test  was  significant,  t  (47)  =  - 1 .83 1 ,  p  <  .  1 0.  The  results  from  the  survey  are  presented 
in  Table  3. 
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Table  3.  Paired  Differences 


Pairs 

Mean 

Std.  Dev 

Std.  Error 
Mean 

Lower 
95%  Cl 

Upper 
95%  Cl 

t 

df 

Sig.  (2-tailed) 

AQ1  -  WQ1 

-.28 

1.748 

.226 

-.73 

.17 

-1.256 

59 

.214 

AQ2  -  WQ2 

-.24 

1.622 

.239 

-.72 

.24 

-1.000 

45 

.323 

AQ3-WQ3 

.00 

1.758 

.235 

-.47 

.47 

.000 

55 

1.000 

AQ4  -  WQ4 

-.07 

1.375 

.179 

-.43 

.29 

-.379 

58 

.706 

AQ5  -  WQ5 

.20 

1.711 

.219 

-.24 

.63 

.898 

60 

.373 

AQ6  -  WQ6 

\45 

1.656 

.217 

.01 

.88 

2.061 

57 

\044 

AQ7-WQ7 

.12 

1.499 

.197 

-.27 

.51 

.613 

57 

.542 

AQ8  -  WQ8 

-.08 

1.426 

.198 

-.47 

.32 

-.389 

51 

.699 

AQ9  -  WQ9 

-.24 

1.648 

.233 

-.71 

.23 

-1.030 

49 

.308 

AQ10-WQ10 

-.15 

1.827 

.238 

-.63 

.32 

-.641 

58 

.524 

AQ 11  -  WQ 1 1 

-.07 

1.745 

.225 

-.52 

.38 

-.296 

59 

.768 

AQ12  -  WQ12 

* 47 

1.665 

.225 

-.92 

-.02 

-2.106 

54 

|  040 

AQ13-WQ13 

-.12 

1.534 

.201 

-.52 

.28 

-.599 

57 

.551 

AQ14-  WQ14 

-.14 

1.515 

.199 

-.54 

.26 

-.693 

57 

.491 

AQ15-WQ15 

.09 

1.478 

.194 

-.30 

.47 

.444 

57 

.659 

AQ16-WQ16 

-.05 

1.594 

.209 

-.47 

.37 

-.247 

57 

.806 

AQ17-WQ17 

-.20 

1.531 

.205 

-.61 

.21 

-.960 

55 

.341 

AQ18-WQ18 

-.16 

1.499 

.200 

-.56 

.24 

-.802 

55 

.426 

AQ19-WQ19 

.00 

1.451 

.191 

-.38 

.38 

.000 

57 

1.000 

AQ20  -  WQ20 

.03 

1.605 

.207 

-.38 

.45 

.161 

59 

.873 

AQ21  -  WQ21 

1 41 

1.376 

.203 

.00 

.82 

2.036 

45 

1 048 

AQ22  -  WQ22 

.66 

1.825 

.258 

.14 

1.18 

2.557 

49 

S  014 

AQ23  -  WQ23 

.26 

1.725 

.252 

-.25 

.76 

1.015 

46 

.316 

AQ24  -  WQ24 

-.52 

2.055 

.261 

-1.04 

.01 

-1.978 

61 

1 052 

AQ3  -  WQ25 

-.36 

2.099 

.273 

-.90 

.19 

-1.303 

58 

.198 

AQ26  -  WQ26 

-.34 

2.298 

.302 

-.95 

.26 

-1.143 

57 

.258 

AQ27  -  WQ27 

.26 

1.259 

.178 

-.10 

.62 

1.461 

49 

.150 

AQ28  -  WQ28 

-.36 

1.223 

.178 

172 

.00 

-2.027 

46 

i  048 

AQ29  -  WQ29 

.17 

1.267 

.174 

-.18 

.52 

.976 

52 

.334 

AQ30  -  WQ30 

-.23 

1.567 

.217 

-.67 

.21 

-1.062 

51 

.293 

AQ31  -WQ31 

-.06 

1.049 

.150 

-.36 

.24 

-.409 

48 

.685 

AQ32  -  WQ32 

-.40 

1.498 

.216 

-.83 

.04 

-1.831 

47 

1 074 

AQ33  -  WQ33 

.00 

1.161 

.169 

-.34 

.34 

.000 

46 

1.000 

AQ34  -  WQ34 

-.14 

1.457 

.222 

-.59 

.31 

-.628 

42 

.533 

AQ35-WQ35 

.00 

1.155 

.176 

-.36 

.36 

.000 

42 

1.000 

AQ36  -  WQ36 

-.02 

1.387 

.217 

-.46 

.41 

-.113 

40 

.911 

Survey  Results 

Overall,  munitions  users  chose  WST  as  the  more  favorable  application.  All  but 
two  construct  measures  supported  the  overall  rating  of  the  application.  Munitions 
personnel  liked  the  screen  layout  of  AMST  better  than  that  of  WST  even  though  WST 
seemed  to  be  the  application  that  better  used  terminology  familiar  to  munitions  personnel. 
WST  was  also  the  easier  of  the  two  applications  to  manipulate  although  the  AMST  help 
features  were  identified  as  the  better  of  the  two.  The  Warrior  Support  Tool  provided 
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information  in  a  timelier  manner  than  did  the  Agile  Munitions  Support  Tool.  The 
importance  and  performance  of  the  features  available  in  each  application  was  also  in 
favor  of  WST.  Table  4  displays  the  results  of  the  survey. 


Table  4.  Survey  Summary  Statistics 


Construct 

AMST  n 

AMST  a 

WST  \x 

WST  a 

Diff  of  Mean 

Overall  Satisfaction  (1-3) 

Ql 

4.50 

1.589 

4.78 

1.180 

-.28 

02 

4.41 

1.641 

4.65 

1.494 

-.24 

03 

4.48 

1.595 

4.48 

1.362 

.00 

Screen  Layout  (4-6) 

Q4 

4.75 

1.625 

4.81 

1.332 

-.07 

Q5 

5.08 

1.666 

4.89 

1.450 

.20 

Q6 

5.34 

1.606 

4.90 

1.471 

.45 

Text  Communication  &  Terminology  (7-9) 

Q7 

5.40 

1.426 

5.28 

1.268 

.12 

Q8 

4.69 

1.591 

4.77 

1.409 

-.08 

Q9 

4.14 

1.414 

4.38 

1.338 

-.24 

Ease  of  Use/Leaming  (10-20) 

-  Learning  (10-12) 

-  Intuitive  (13-15) 

-Presentation  (16-18) 

-  Novice  Friendly  (19-20) 

Q10 

4.44 

1.695 

4.59 

1.452 

-.15 

Qll 

4.78 

1.833 

4.85 

1.471 

-.07 

Q12 

4.09 

1.567 

4.56 

1.512 

-.47 

Q13 

4.50 

1.625 

4.62 

1.531 

-.12 

Q14 

4.67 

1.560 

4.81 

1.444 

-.14 

Q15 

4.66 

1.573 

4.57 

1.476 

.09 

Q16 

4.60 

1.533 

4.66 

1.433 

-.05 

017 

4.55 

1.572 

4.75 

1.430 

-.20 

Q18 

4.27 

1.578 

4.43 

1.425 

-.16 

Q19 

4.26 

1.639 

4.26 

1.562 

.00 

Q20 

4.27 

1.517 

4.23 

1.630 

.03 

Online  Help  (21-23) 

Q21 

4.85 

1.660 

4.43 

1.615 

.41 

Q22 

5.26 

1.626 

4.60 

1.629 

.66 

Q23 

4.60 

1.570 

4.34 

1.632 

.26 

Speed  (24-26) 

Q24 

4.42 

1.816 

4.94 

1.329 

-.52 

Q25 

4.22 

1.811 

4.58 

1.522 

-.36 

Q26 

4.21 

1.971 

4.55 

1.569 

-.34 

Importance  of  Features  (27-31) 

Q27 

5.32 

1.377 

5.06 

1.504 

.26 

Q28 

4.62 

1.423 

4.98 

1.375 

-.36 

Q29 

5.45 

1.338 

5.28 

1.536 

.17 

Q30 

4.79 

1.473 

5.02 

1.407 

-.23 

Q31 

5.37 

1.334 

5.43 

1.486 

-.06 

Performance  of  Features  (32-36) 

032 

4.69 

1.504 

5.08 

1.499 

-.40 

Q33 

5.28 

1.394 

5.28 

1.542 

.00 

Q34 

4.74 

1.399 

4.88 

1.159 

-.14 

Q35 

5.14 

1.320 

5.14 

1.597 

.00 

Q36 

4.66 

1.296 

4.68 

1.422 

-.02 
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The  survey  participants  were  also  asked  to  identify  any  features  they  thought 
should  be  added  to  the  applications  to  enable  them  to  better  meet  the  needs  of  the 
munitions  community  and  to  rate  their  overall  experience  with  each  application.  In 
general,  all  users  were  satisfied  with  the  applications.  No  one  was  “awed”  by  the 
capabilities  of  the  application  nor  were  they  completely  disenchanted  with  the 
capabilities.  Appendix  C  provides  the  comments  that  were  made  in  regards  to  the 
applications. 

Open  forum  interviews  were  also  conducted  after  the  surveying  period  to  identify 
any  additional  concerns  or  thoughts  about  the  applications.  An  issue  that  cannot  go 
unmentioned  is  that  a  noticeable  pattern  emerged  in  the  interviews.  Senior  level 
munitions  personnel  favored  the  information  provided  by  the  Agile  Munitions  Support 
Tool  over  the  Warrior  Support  Tool  and  the  exact  opposite  was  true  for  junior  level 
munitions  personnel.  Senior  level  munitions  personnel  (MSgt  -  Capt)  gave  mention  to 
the  fact  that  AMST  provided  the  exact  information  they  needed  to  keep  mission 
commanders  informed  during  contingency  operations,  in  transit  information.  Junior  level 
munitions  personnel  felt  that  WST  was  useful  at  the  craftsman  level  because  of  the 
illustrations  and  DODIC  information.  Significant  analytical  patterns  emerged  to  support 
the  statements  in  regards  to  specific  information  but  not  for  the  applications  overall.  The 
differences  were  noticed  in  questions  3-5,  8-13,  15,  19,  and  21for  AMST  and  questions  2- 
4,  9-19,  21,  24,  and  26  for  WST.  Table  6  shows  the  mean  ratings  for  the  questions  as 
well  as  the  significant  level.  The  complete  table  can  be  viewed  in  Appendix  D  and  the 
specific  questions  can  be  referenced  in  Appendix  A. 
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Table  5  ANOVA  Results 


Question 

Group 

Mean 

Sig. 

AQ3 

Jr./Sr. 

5.03/4.04 

.015 

AQ4 

AQ5 

Jr./Sr. 

Jr./Sr. 

5.34/4.11 

5.53/4.63 

.002 

.034 

AQ8 

AQ9 

Jr./Sr. 

Jr./Sr. 

5.23/4.13 

4.64/3.67 

.006 

.012 

AQ10 
AQ1 1 
AQ12 
AQ13 
AQ15 

Jr./Sr. 

Jr./Sr. 

Jr./Sr. 

Jr./Sr. 

Jr./Sr. 

4.85/3.70 

5.22/4.11 

4.47/3.54 

5.00/3.81 

5.17/4.15 

.010 

.017 

.022 

.004 

.012 

AQ19 

AQ21 

Jr./Sr. 

Jr./Sr. 

4.77/3.62 

5.19/4.29 

.006 

.043 

Question 

Group 

Mean 

Sig. 

WQ2 

Jr./Sr. 

5.11/4.12 

.007 

WQ3 

Jr./Sr. 

4.89/4.00 

.012 

WQ4 

Jr./Sr. 

5.11/4.42 

.039 

WQ9 

Jr./Sr. 

4.75/3.91 

.020 

WQ10 

Jr./Sr. 

5.00/4.08 

.011 

WQ11 

Jr./Sr. 

5.23/4.35 

.018 

WQ12 

Jr./Sr. 

5.00/4.00 

.010 

WQ13 

Jr./Sr. 

5.14/4.00 

.003 

WQ14 

Jr./Sr. 

5.17/4.33 

.026 

WQ15 

Jr./Sr. 

5.15/3.80 

.000 

WQ16 

Jr./Sr. 

5.03/4.30 

.042 

WQ17 

Jr./Sr. 

5.06/4.35 

.047 

WQ18 

Jr./Sr. 

4.91/3.88 

.004 

WQ26 

Jr./Sr. 

4.97/3.96 

.011 

WQ19 

Jr./Sr. 

4.67/3.73 

.015 

WQ21 

Jr./Sr. 

5.00/4.00 

.027 

WQ24 

Jr./Sr. 

5.25/4.52 

.028 

On  the  surveys,  questions  1  through  3  referred  to  the  overall  rating  of  the 


applications.  Questions  4  through  6  aimed  to  identify  which  application  had  the  better 


screen  layout.  Questions  7  through  9  considered  the  communication  and  terminology  of 


the  applications.  Questions  10  through  18  as  well  as  25  and  26  referred  to  the  ease  of 


using  the  applications.  Questions  19  through  21  focused  on  the  help  features  of  the 


applications  and  22  through  24  attempted  to  get  a  rating  on  the  speed  and  functionality. 


How  supportable  is  each  application  in  terms  of  information  needs  versus  existing 

systems? 

Based  on  the  results  of  the  survey  and  interviews  with  developers,  both 
applications  are  very  supportable  in  tenns  of  information  needs  versus  existing  systems. 
Both  applications  receive  munitions  information  from  the  legacy  systems  that  are 
currently  in  use  in  the  munitions  community.  The  applications  only  show  the  infonnation 
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that  is  fed  to  them  in  a  more  user-friendly  nature.  Two  major  concerns  that  were 
identified  in  terms  of  the  supportability  of  the  applications  were: 

1 .  Security  measures  for  the  applications  need  to  b 

e  addressed  since  they  are  both  web-based  applications  that  display 

sensitive  infonnation. 

2.  Some  sort  of  training  class  or  program  needs  to  be  developed  to  assist 

users  in  manipulating  the  applications  more  effectively. 

With  both  applications  receiving  information  from  legacy  systems,  the  user  needs 
to  be  aware  of  the  garbage  in  garbage  out  syndrome.  The  accuracy  of  the  information 
received  on  either  application  is  only  as  good  as  the  individual  putting  the  original 
information  in  one  of  the  given  legacy  systems. 

Summary 

In  this  chapter,  the  comparative  results  between  the  Warrior  Support  Tool  and  the 
Agile  Munitions  Support  Tool  were  presented.  Discussion  as  to  the  ideal  application  was 
also  presented  as  well  as  suggestions  for  further  research.  Munitions  users  detennined 
that  a  useful  munitions  IT  application  would  track  assets  and  expenditures,  identify 
munitions  shortages  and  availability,  and  provide  requisition  history  information.  WST 
was  able  to  track  assets  and  expenditures,  identify  munitions  shortages  and  availability, 
and  provide  simulation  capabilities  and  aircraft  compatibility  features.  AMST  was  able 
to  track  assets  and  expenditures,  identify  munitions  shortages  and  availability,  provide 
requisition  history  infonnation,  and  track  ISIR  information.  Overall,  each  application 
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met  users’  needs  and  was  supportable  versus  existing  systems  because  they  are  fed  by  the 
existing  legacy  systems  however,  WST  was  chosen  as  the  more  favorable  application.  In 
the  following  chapter,  the  research  questions  will  be  discussed  a  bit  further  to  provide 
insight  as  to  the  driving  force  behind  this  research. 
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V.  Conclusion  and  Recommendations 


Overview 

This  chapter  discusses  the  results  of  the  statistical  analyses  performed  in  Chapter 
IV.  The  analyses  are  discussed  in  reference  to  the  investigative  questions  that  were  posed 
and  conclusions  regarding  this  research  are  drawn.  Additionally,  this  chapter  discusses 
the  limitations  of  the  research  as  well  as  the  theoretical  and  practical  implications  of  the 
research  results.  The  final  section  of  this  chapter  suggests  further  research  focusing  on 
the  munitions  information  technology  applications  that  were  studied. 


Summary  of  Findings 

What  are  the  customer  needs? 

126  specific  needs  were  identified  from  the  analysis  of  the  documents,  which 
were  eventually  categorized  under  6  major  headings.  For  an  IT  application  to  be  useful  in 
the  munitions  community,  it  had  to: 

1 .  Track  munitions  assets  and  all  up  rounds. 

2.  Track  munitions  expenditures. 

3.  Identify  munitions  shortages. 

4.  Identify  available  munitions. 

5.  Track  in  transit  assets. 

6.  Provide  requisition  history  for  assets. 
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What  does  each  application  provide? 

The  application  capabilities  were  identified  through  the  use  of  presentations  and 
interviews  with  system  developers,  planning  documents  available  via  the  web,  literature 
devoted  to  WST  and  AMST  application  capabilities,  and  manipulation  of  the 
applications.  A  survey  was  also  developed  for  munitions  personnel  to  identify  and/or 
validate  the  capabilities  and  ease  of  manipulating  the  applications.  It  was  found  that 
WST  is  capable  of  providing  asset  locations,  balances,  condition  codes,  expenditure  rates, 
asset  and  aircraft  compatibility,  and  simulation  capabilities.  The  AMST  application  was 
found  to  be  able  to  provide  asset  locations,  balances,  expenditure  rates,  simulation 
capabilities,  asset  requisition  history  information,  in  transit  asset  visibility,  and  ISIR 
information. 

How  well  does  each  application  meet  the  needs? 

WST  does  not  adequately  track  in  transit  assets  or  provide  requisition  history. 
However,  the  Warrior  Support  Tool  received  praise  on  all  of  the  other  features  it 
provided.  As  for  the  requisition  history  feature,  the  application  did  not  directly  provide 
an  in-depth  report  of  requisition  history  but  the  information  can  be  obtained  through 
reverse  engineering  and  deductive  reasoning.  One  must  be  aware  that  by  obtaining  the 
information  through  reverse  engineering,  miscalculations  can  be  made. 

AMST  also  received  excellent  praise  but  concern  was  expressed  as  to  the  speed  of 
the  application  and  the  fact  that  too  much  infonnation  had  to  be  manually  input  into  the 
application.  Munitions  users  preferred  the  point  and  click  features  that  were  available 
through  the  WST  application  over  the  NSN  input  requirement  of  the  AMST  application. 
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Extreme  concern  was  expressed  as  to  the  level  of  security  provided  for  both 
applications.  Munitions  personnel  acknowledged  the  understanding  that  CAS  was  being 
declassified  but  were  concerned  with  the  fact  that  the  available  munitions  infonnation 
was  still  sensitive  in  nature. 

Based  on  these  inputs,  the  following  recommendations  for  action  and/or  further 
research  were  presented  to  rectify  the  concerns. 

1 .  Investigate  the  requirements  and  feasibility  of  incorporating  an  in  transit  tracking 
feature  and  requisition  history  feature  in  the  Warrior  Support  Tool. 

2.  Investigate  avenues  available  to  limit  the  amount  of  information  input  required  by  the 
user  to  manipulate  the  Agile  Munitions  Support  Tool. 

3.  Conduct  an  in-depth  study  into  the  required  hardware  needed  to  effectively  manipulate 
the  applications  to  ensure  availability  of  timely  information. 

4.  Conduct  an  in-depth  study  into  the  security  issues  surrounding  military  sensitive 
information  and  the  use  of  web-based  IT  applications. 

How  supportable  is  each  application  in  terms  of  information  needs  versus 
existing  systems? 

Based  on  the  results  of  the  survey  and  interviews  with  developers,  both 
applications  are  very  supportable  in  tenns  of  information  needs  versus  existing  systems. 
Both  applications  receive  munitions  information  from  the  legacy  systems  that  are 
currently  in  use  in  the  munitions  community.  The  applications  only  show  the  infonnation 
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that  is  fed  to  them  in  a  more  user-friendly  nature.  Two  major  concerns  that  were 
identified  in  terms  of  the  supportability  of  the  applications  were: 

1 .  Security  measures  for  the  applications  need  to  be  addressed  since  they 

are  both  web-based  application  that  display  sensitive  information. 

2.  Some  sort  of  training  class  or  program  needs  to  be  developed  to  assist 

users  in  manipulating  the  applications  more  effectively. 

With  both  applications  receiving  information  from  legacy  systems,  the  user  needs 
to  be  aware  of  the  garbage  in  garbage  out  syndrome.  The  accuracy  of  the  information 
received  from  either  application  is  only  as  good  as  the  individual  putting  the  original 
information  in  one  of  the  given  legacy  systems. 

As  to  the  objective  of  identifying  if  one  IT  application  will  suffice  in  the 
munitions  community,  the  answer  is  yes  but  a  determination  as  to  which  application 
should  be  used  needs  to  be  determined  through  further  analysis.  Cost  analysis  and  more 
in-depth  field-testing  should  be  administered  before  such  a  decision  is  made.  This 
research  only  lays  a  foundation  for  the  development  of  such  future  research  endeavors. 

Limitations 

It  quickly  became  apparent  that  the  most  significant  limitation  of  this  research 
was  that  the  IT  applications  that  were  being  compared  in  this  study  were  new  to  the 
munitions  community,  which  limited  the  amount  of  valuable  user  knowledge  available 
for  the  research.  As  such,  subject  matter  experts  were  used  almost  exclusively  to 
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facilitate  an  accurate  understanding  of  the  operational  aspects  of  both  applications.  The 
subject  matter  experts  were  mostly  developers  that  were  responsible  for  fielding  the 
applications,  which  brought  about  another  limitation  in  that  bias  could  have  crept  in  to 
the  study.  A  system  developer  will  naturally  favor  the  system  or  application  that  he  or 
she  is  developing. 

Another  limitation  emerged  during  the  surveying  sessions.  An  unbiased  scenario 
for  the  operational  testing  of  both  applications  was  created  as  well  as  an  adequate 
understanding  of  the  operational  aspects  of  each  application.  However,  the  computer  lab 
that  was  used  experienced  a  number  of  computer  problems  during  the  application 
manipulation  sessions.  Gateway  Pentium  3  computers  were  in  use  but  many  of  them 
locked  up  or  timed  out  when  munitions  users  were  attempting  to  retrieve  information.  To 
overcome  the  frustration  that  some  of  the  sampling  population  may  have  experienced, 
actual  users  of  the  applications  were  going  to  be  sampled.  However,  with  the 
applications  being  web-based,  it  was  hard  to  obtain  records  of  actual  users  of  the  system. 
The  number  of  accounts  that  were  awarded  was  known  but  the  frequency  of  use  was  not. 
That  aspect  made  identifying  actual  “full-time”  users  almost  impossible. 

Comments  accompanying  the  survey  responses  suggested  another  limitation.  The 
sampling  population  could  have  been  a  bit  larger  because  a  learning  curve  may  have  been 
present  in  that  computer  literate  people  may  have  pick  up  on  the  manipulation  of  the 
applications  quicker  than  those  that  did  not  spend  a  great  deal  of  time  on  computers. 
Everyone  was  asked  to  perfonn  the  same  tasks,  however,  there  may  have  been  munitions 
users  in  the  sampling  population  that  did  not  know  how  to  perform  a  given  task. 
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Implications 


The  1990s  began  the  surge  in  the  growth  of  information  technology.  Many 
creative  ways  of  collecting  and  managing  infonnation  were  established  to  provide  useful 
data  to  decision  makers  to  improve  the  decision  making  process.  A  great  deal  of  time 
and  effort  is  spent  manually  pulling,  compiling,  and  analyzing  munitions  data  to  provide 
meaningful  information  to  facilitate  the  decision  making  process.  A  munitions  IT 
application  that  can  automatically  pull  munitions  data  from  existing  legacy  ammunition 
information  systems  and  combine  that  data  to  paint  a  complete  picture  for  the  user  will 
provide  a  noticeable  edge  to  the  flexibility  of  administering  munitions  power.  Such  a 
technology  will  provide  commanders  and  munitions  users  the  tools  and  information 
necessary  to  make  the  timely  and  proactive  decisions  necessary  to  keep  our  United  States 
military  poised  to  respond  to  world  crises.  The  benefits  would  be  the  quick  availability 
of  reliable  information  whenever  it  was  needed;  the  elimination  of  separate  records  or 
other  local  documents,  along  with  the  required  time  to  maintain  them;  and  the 
maintainability  of  accurate,  timely  information  in  munitions  legacy  databases. 

Recommendations  for  Action/Further  Research 

The  recommendations  for  further  research  involve  security  measures  and  Internet 
capabilities.  First,  WST  and  AMST  will  display  information  contained  in  CAS  over  the 
Internet.  Although  munitions  information  in  CAS  has  been  declassified,  it  is  still 
sensitive.  Munitions  planners  and  users  need  to  understand  the  security  features  and 
measures  of  the  two  applications  as  well  as  the  plan  and  reliability  of  the  applications  to 
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protect  sensitive  munitions  infonnation.  Specifically,  they  must  determine  how  well 
WST  and  AMST  provide  necessary  security  measures  for  the  protection  of  sensitive 
munitions  data  in  order  to  meet  the  needs  of  the  munitions  community.  A  few 
investigative  questions  that  may  lead  to  an  answer  are: 

1 .  What  are  the  security  measures  incorporated  in  WST  and  AMST? 

a)  How  do  the  security  features  operate? 

b)  What  backup  procedures  are  available  to  meet  security  requirements? 

2.  How  well  do  WST  and  AMST  guard  against  hacking? 

a)  What  security  guards  have  been  developed? 

b)  How  reliable  is  the  up  time  of  the  security  guards? 

3.  How  can  the  applications  be  improved  to  better  provide  secure  munitions  information? 

Secondly,  munitions  planners  and  users  need  to  understand  the  contingency  plan 
to  provide  needed  munitions  infonnation  under  austere  working  conditions  where 
Internet  connections  are  not  available.  In  particular,  they  must  detennine  how  to  provide 
web-based  munitions  information  to  locations  that  do  not  have  access  to  fiber  optic 
cables  for  Internet  connectivity.  Some  investigative  questions  that  may  lead  to  an  answer 
are  as  follows: 

1 .  What  contingency  plans  or  equipment  have  been  developed  to  enable  successful 
manipulation  of  the  WST  and  AMST  applications  in  austere  environments? 

a)  How  detailed  is  the  contingency  plan? 

b)  What  is  the  cost  of  additional  equipment  to  support  operational  needs? 
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2.  How  well  do  WST  and  AMST  operate  in  austere  conditions? 

a)  Are  there  significant  differences  in  the  performance  time  of  the  systems? 

b)  Are  security  measures  compromised? 

3.  How  can  the  applications  be  improved  to  better  provide  munitions  information  in 
austere  working  conditions? 

Finally,  an  addition  to  this  comparative  study  that  could  warrant  further  research 
is  an  analysis  of  each  application  by  expert  users  of  the  opposite  application. 
Specifically,  have  a  fluent  user  of  WST  manipulate  the  AMST  application  and  AMST 
users  manipulate  WST  to  determine  how  easy  and  user  friendly  the  applications  are. 
Additionally,  they  should  truly  be  able  to  identify  how  well  each  application  meets  the 
needs  of  the  munitions  community. 


64 


APPENDIX  A 
QUESTIONAIRES 
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AFIT -2 002-09- W  ST 


Purpose:  The  purpose  of  this  research  is  to  investigate  the  ability  of  the  Warrior  Support  Tool  (WST)  to 
meet  the  needs  of  the  munitions  community. 

Participation:  You  are  one  of  several  munitions  subject  matter  experts  asked  to  participate  in  this 
research.  Thank  you  in  advance  for  participating — your  answers  are  very  important. 

Confidentiality:  All  answers  are  anonymous.  No  identification  of  individual  responses  will  occur. 
Privacy  Notice 


In  accordance  with  AFI  37-132,  Paragraph  3.2,  the  following  information  is  provided  as  required  by  the 
Privacy  Act  of  1974: 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation  by;  implemented  by 
AFI  36-2601,  Air  Force  Personnel  Survey  Program. 

Purpose:  To  obtain  information  regarding  the  capabilities  of  WST. 

Routine  Use:  No  analysis  of  individual  responses  will  be  conducted  and  only  members  of  the  research 
team  will  be  permitted  access  to  the  raw  data. 

Participation:  Participation  is  VOLUNTARY.  No  adverse  action  will  be  taken  against  any  member  who 
does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  the  survey. 
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This  survey  is  designed  to  determine  the  ability  of  the  WST  to  meet  the  needs  of  the  munitions  community. 


SECTION  2:  User  Profile: 

Sex  {please  check  one):  _ Male  _ Female 

Duty  Title: _ 

AFSC: 


MAJCOM: 

Rank: 


Years  in  AFSC: 


Flow  much  previous  experience  do  you  have  with  the  WST  application? 
In  a  typical  week,  how  much  time  do  you  spend  on  a  computer? 

In  a  typical  week,  how  much  time  do  you  spend  with  CAS? 

Rate  your  experience  with  computer  applications. 


novice 


hours 

hours 

hours 

expert 


1  2  3  4  5  6  7 


Briefly  describe  the  sorts  of  tasks  you  perform  on  computers  in  general. 


SECTION  2:  General  Knowledge  Level: 

Please  rate  the  series  of  questions  based  on  your  perceived  knowledge  level  of  each  category.  Please 
circle  the  numbers  that  most  appropriately  reflect  your  impressions  about  using  WST.  Not 
Applicable  =  N/A 


1. 

Novices  can  accomplish  tasks  knowing  only  a  few  commands. 

difficultly 

easily 

1  2  3 

4 

5 

6  7 

n/a 

2. 

Time  it  takes  WST  to  respond  during  operations. 

slow 

fast 

1  2  3 

4 

5 

6  7 

n/a 

3. 

Accessing  help  messages. 

difficult 

easy 

1  2  3 

4 

5 

6  7 

n/a 

4. 

Task  completion  follows  a  logical  sequence. 

rarely 

always 

1  2  3 

4 

5 

6  7 

n/a 

5. 

How  would  you  rate  operating  with  WST? 

frustrating 

satisfying 

1  2  3 

4 

5 

6  7 

n/a 

6. 

Getting  started  with  WST. 

difficult 

easy 

1  2  3 

4 

5 

6  7 

n/a 

7. 

Arrangement  of  information  on  screen. 

illogical 

logical 

1  2  3 

4 

5 

6  7 

n/a 
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8.  ] 

Messages  that  appear  on  the  screen. 

confusing 

clear 

1  2 

3 

4 

5 

6  7 

n/a 

9.  ] 

Exploration  of  WST  features. 

risky 

safe 

1  2 

3 

4 

5 

6  7 

n/a 

10. 

Are  terms  related  to  munitions  operations? 

unrelated 

related 

1  2 

3 

4 

5 

6  7 

n/a 

11. 

Speed  of  application. 

slow 

fast 

1  2 

3 

4 

5 

6  7 

n/a 

12. 

Help  messages. 

confusing 

clear 

1  2 

3 

4 

5 

6  7 

n/a 

13. 

Is  task  performance  straightforward? 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

14. 

Are  needs  of  experienced  and  inexperienced  users  considered? 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

15. 

Overall,  how  would  you  characterize  WST? 

difficult 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

16. 

Learning  to  operate  WST. 

difficult 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

17. 

Exploration  of  WST  features  by  trial  and  error. 

discouraging 

encouraging 

1  2 

3 

4 

5 

6  7 

n/a 

18. 

Were  the  screen  layouts  helpful? 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

19. 

Time  it  takes  WST  to  display  information. 

slow 

fast 

1  2 

3 

4 

5 

6  7 

n/a 

20. 

How  would  you  rate  the  performance  of  WST? 

rigid 

flexible 

1  2 

3 

4 

5 

6  7 

n/a 

21. 

Going  back  to  the  previous  screen. 

impossible 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

22. 

Discovering  new  features  in  WST. 

difficult 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

23. 

Users  can  control  amount  of  feedback 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

24. 

Completion  of  a  sequence  of  steps. 

unclear 

clear 

1  2 

3 

4 

5 

6  7 

n/a 

25. 

Time  to  learn  WST. 

slow 

fast 

1  2 

3 

4 

5 

6  7 

n/a 
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26. 

Quality  of  help  messages. 

inadequate 

adequate 

1  2  3 

4 

5 

6  7 

n/a 

27. 

How  important  is  the  asset  location  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

28. 

Rate  the  performance  of  the  asset  location  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

29. 

How  important  is  the  asset  balance  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

30. 

Rate  the  performance  of  the  asset  balance  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

31. 

How  important  is  the  condition  code  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

32. 

Rate  the  performance  of  the  condition  code  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

33. 

How  important  is  the  expenditure  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

34. 

Rate  the  performance  of  the  expenditure  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

35. 

How  important  is  the  compatibility  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

36. 

Rate  the  performance  of  the  compatibility  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

37.  Identify  any  features  that  you  feel  should  be  incorporated  in  the  WST  application. 


38.  Overall,  how  would  you  rate  your  experience  with  WST? 
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AFIT-2002-09-AMST 


Purpose:  The  purpose  of  this  research  is  to  investigate  the  ability  of  the  Agile  Munitions  Support  Tool 
(AMST)  to  meet  the  needs  of  the  munitions  community. 

Participation:  You  are  one  of  several  munitions  subject  matter  experts  asked  to  participate  in  this 
research.  Thank  you  in  advance  for  participating — your  answers  are  very  important. 

Confidentiality:  All  answers  are  anonymous.  No  identification  of  individual  responses  will  occur. 
Privacy  Notice 


In  accordance  with  AFI  37-132,  Paragraph  3.2,  the  following  information  is  provided  as  required  by  the 
Privacy  Act  of  1974: 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation  by;  implemented  by 
AFI  36-2601,  Air  Force  Personnel  Survey  Program. 

Purpose:  To  obtain  information  regarding  the  capabilities  of  AMST. 

Routine  Use:  No  analysis  of  individual  responses  will  be  conducted  and  only  members  of  the  research 
team  will  be  permitted  access  to  the  raw  data. 

Participation:  Participation  is  VOLUNTARY.  No  adverse  action  will  be  taken  against  any  member  who 
does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  the  survey. 
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This  survey  is  designed  to  determine  the  ability  of  the  AMST  to  meet  the  needs  of  the  munitions 
community. _ 


SECTION  2:  User  Profile: 

Sex  (please  check  one):  _ Male  _ Female  MAJCOM: _ 

Duty  Title: _  Rank: _ 

AFSC: _  Years  in  AFSC: _ 

Flow  much  previous  experience  do  you  have  with  the  AMST  application? 

In  a  typical  week,  how  much  time  do  you  spend  on  a  computer? 

In  a  typical  week,  how  much  time  do  you  spend  with  CAS? 

Rate  your  experience  with  computer  applications.  novice 


hours 

hours 

hours 

expert 


1  2  3  4  5  6  7 


Briefly  describe  the  sorts  of  tasks  you  perform  on  computers  in  general. 


SECTION  2:  General  Knowledge  Level: 

Please  rate  the  series  of  questions  based  on  your  perceived  knowledge  level  of  each  category.  Please 
circle  the  numbers  that  most  appropriately  reflect  your  impressions  about  using  AMST.  Not 
Applicable  =  N/A 


1. 

Novices  can  accomplish  tasks  knowing  only  a  few  commands. 

difficultly 

easily 

1  2  3 

4 

5 

6  7 

n/a 

2. 

Time  it  takes  AMST  to  respond  during  operations. 

slow 

fast 

1  2  3 

4 

5 

6  7 

n/a 

3. 

Accessing  help  messages. 

difficult 

easy 

1  2  3 

4 

5 

6  7 

n/a 

4. 

Task  completion  follows  a  logical  sequence. 

rarely 

always 

1  2  3 

4 

5 

6  7 

n/a 

5. 

How  would  you  rate  operating  with  AMST? 

frustrating 

satisfying 

1  2  3 

4 

5 

6  7 

n/a 

6. 

Getting  started  with  AMST. 

difficult 

easy 

1  2  3 

4 

5 

6  7 

n/a 
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7.  Arrangement  of  information  on  screen. 

illogical 

logical 

1  2 

3 

4 

5 

6  7 

n/a 

8.  Messages  that  appear  on  the  screen. 

confusing 

clear 

1  2 

3 

4 

5 

6  7 

n/a 

9.  Exploration  of  AMST  features. 

risky 

safe 

1  2 

3 

4 

5 

6  7 

n/a 

10.  Are  terms  related  to  munitions  operations? 

unrelated 

related 

1  2 

3 

4 

5 

6  7 

n/a 

1 1 .  Speed  of  application. 

slow 

fast 

1  2 

3 

4 

5 

6  7 

n/a 

12.  Help  messages. 

confusing 

clear 

1  2 

3 

4 

5 

6  7 

n/a 

13.  Is  task  performance  straightforward? 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

14.  Are  needs  of  experienced  and  inexperienced  users  considered? 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

15.  Overall,  how  would  you  characterize  AMST? 

difficult 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

16.  Learning  to  operate  AMST. 

difficult 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

17.  Exploration  of  WST  features  by  trial  and  error. 

discouraging 

encouraging 

1  2 

3 

4 

5 

6  7 

n/a 

18.  Were  the  screen  layouts  helpful? 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

19.  Time  it  takes  WST  to  display  information. 

slow 

fast 

1  2 

3 

4 

5 

6  7 

n/a 

20.  How  would  you  rate  the  performance  of  AMST? 

rigid 

flexible 

1  2 

3 

4 

5 

6  7 

n/a 

21.  Going  back  to  the  previous  screen. 

impossible 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

22.  Discovering  new  features  in  AMST. 

difficult 

easy 

1  2 

3 

4 

5 

6  7 

n/a 

23.  Users  can  control  amount  of  feedback 

never 

always 

1  2 

3 

4 

5 

6  7 

n/a 

24.  Completion  of  a  sequence  of  steps. 

unclear 

clear 

1  2 

3 

4 

5 

6  7 

n/a 
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25. 

Time  to  learn  AMST. 

slow 

fast 

1  2  3 

4 

5 

6  7 

n/a 

26. 

Quality  of  help  messages. 

inadequate 

adequate 

1  2  3 

4 

5 

6  7 

n/a 

27. 

How  important  is  the  intransit  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

28. 

Rate  the  performance  of  the  intransit  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

29. 

How  important  is  the  asset  balance  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

30. 

Rate  the  performance  of  the  asset  balance  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

31. 

How  important  is  the  expenditure  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

32. 

Rate  the  performance  of  the  expenditure  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

33. 

How  important  is  the  requisition  history  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

34. 

Rate  the  performance  of  the  requisition  history  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

35. 

How  important  is  the  ISIR  information  feature? 

unimportant 

important 

1  2  3 

4 

5 

6  7 

n/a 

36. 

Rate  the  performance  of  the  ISIR  information  feature. 

marginal 

outstanding 

1  2  3 

4 

5 

6  7 

n/a 

37.  Identify  any  features  that  you  feel  should  be  incorporated  in  the  AMST  application. 


38.  Overall,  how  would  you  rate  your  experience  with  AMST? 
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WST  Factors 
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WQ1 

WQ2 

WQ3 

WQ4 

WQ5 

WQ6 

WQ7 

WQ8 

WQ9 

WQ10 

WQ11 

WQ12 

WQ13 

WQ14 

WQ15 

WQ16 

WQ17 

WQ18 

WQ1 

I'M 

0.755 

0.776 

0.754 

0.681 

0.632 

0.727 

0.681 

0.794 

0.808 

0.783 

0.678 

0.731 

0.717 

0.699 

0.751 

0.700 

0.655 

WQ2 

0.755 

ijoS 

0.854 

0.655 

0.716 

0.835 

0.568 

0.837 

0.829 

0.851 

0.833 

0.789 

0.816 

0.889 

0.827 

0.806 

0.712 

0.807 

WQ3 

0.776 

0.854 

0.707 

0.596 

0.728 

0.588 

0.743 

0.817 

0.798 

0.811 

0.731 

0.778 

0.812 

0.807 

0.791 

0.687 

0.759 

WQ4 

0.754 

0.655 

0.707 

Mi 

0.742 

0.644 

0.765 

0.698 

0.675 

0.674 

0.651 

0.693 

0.680 

0.657 

0.767 

0.769 

0.732 

0.706 

WQ5 

0.681 

0.716 

0.596 

0.742 

MM 

0.723 

0.665 

0.799 

0.601 

0.655 

0.643 

0.714 

0.645 

0.645 

0.675 

0.744 

0.848 

0.797 

WQ6 

0.632 

0.835 

0.728 

0.644 

0.723 

i.ooo 

0.378 

0.806 

0.747 

0.757 

0.804 

0.710 

0.746 

0.784 

0.755 

0.699 

0.631 

0.726 

WQ7 

0.727 

0.568 

0.588 

0.765 

0.665 

0.378 

PM 

0.561 

0.588 

0.555 

0.533 

0.570 

0.543 

0.528 

0.568 

0.705 

0.720 

0.650 

WQ8 

0.681 

0.837 

0.743 

0.698 

0.799 

0.806 

0.561 

0.739 

0.718 

0.726 

0.801 

0.779 

0.742 

0.725 

0.806 

0.793 

0.859 

WQ9 

0.794 

0.829 

0.817 

0.675 

0.601 

0.747 

0.588 

0.739 

IMS 

0.837 

0.857 

0.689 

0.727 

0.813 

0.824 

0.735 

0.603 

0.733 

WQ10 

0.808 

0.851 

0.798 

0.674 

0.655 

0.757 

0.555 

0.718 

0.837 

0.946 

0.718 

0.821 

0.858 

0.806 

0.800 

0.693 

0.705 

WQ11 

0.783 

0.833 

0.811 

0.651 

0.643 

0.804 

0.533 

0.726 

0.857 

0.946 

m 

0.722 

0.844 

0.873 

0.814 

0.831 

0.679 

0.747 

WQ12 

0.678 

0.789 

0.731 

0.693 

0.714 

0.710 

0.570 

0.801 

0.689 

0.718 

0.722 

»« 

0.845 

0.826 

0.813 

0.832 

0.776 

0.853 

WQ13 

0.731 

0.816 

0.778 

0.680 

0.645 

0.746 

0.543 

0.779 

0.727 

0.821 

0.844 

0.845 

1>M 

0.871 

0.875 

0.876 

0.714 

0.810 

WQ14 

0.717 

0.889 

0.812 

0.657 

0.645 

0.784 

0.528 

0.742 

0.813 

0.858 

0.873 

0.826 

0.871 

IM 

0.865 

0.819 

0.669 

0.797 

WQ15 

0.699 

0.827 

0.807 

0.767 

0.675 

0.755 

0.568 

0.725 

0.824 

0.806 

0.814 

0.813 

0.875 

0.865 

in 

0.848 

0.652 

0.841 

WQ16 

0.751 

0.806 

0.791 

0.769 

0.744 

0.699 

0.705 

0.806 

0.735 

0.800 

0.831 

0.832 

0.876 

0.819 

0.848 

MS 

0.822 

0.847 

WQ17 

0.700 

0.712 

0.687 

0.732 

0.848 

0.631 

0.720 

0.793 

0.603 

0.693 

0.679 

0.776 

0.714 

0.669 

0.652 

0.822 

MM 

0.820 

WQ18 

0.655 

0.807 

0.759 

0.706 

0.797 

0.726 

0.650 

0.859 

0.733 

0.705 

0.747 

0.853 

0.810 

0.797 

0.841 

0.847 

0.820 

1.0(18 

WQ19 

0.528 

0.727 

0.635 

0.540 

0.547 

0.526 

0.495 

0.674 

0.599 

0.629 

0.634 

0.726 

0.635 

0.751 

0.623 

0.598 

0.657 

0.785 

WQ20 

0.695 

0.831 

0.699 

0.571 

0.605 

0.634 

0.496 

0.696 

0.689 

0.739 

0.744 

0.717 

0.664 

0.734 

0.621 

0.645 

0.652 

0.699 

WQ21 

0.523 

0.748 

0.578 

0.659 

0.678 

0.688 

0.517 

0.784 

0.577 

0.670 

0.686 

0.781 

0.788 

0.731 

0.733 

0.729 

0.749 

0.807 

WQ22 

0.563 

0.751 

0.620 

0.519 

0.652 

0.596 

0.527 

0.719 

0.627 

0.665 

0.670 

0.679 

0.532 

0.672 

0.602 

0.622 

0.655 

0.718 

WQ23 

0.573 

0.739 

0.664 

0.549 

0.659 

0.655 

0.564 

0.793 

0.677 

0.689 

0.716 

0.747 

0.610 

0.674 

0.626 

0.682 

0.690 

0.761 

WQ24 

0.400 

0.614 

0.579 

0.433 

0.333 

0.695 

0.306 

0.511 

0.592 

0.613 

0.654 

0.389 

0.583 

0.722 

0.566 

0.555 

0.339 

0.457 

WQ25 

0.558 

0.650 

0.598 

0.518 

0.429 

0.505 

0.524 

0.480 

0.585 

0.639 

0.658 

0.558 

0.641 

0.696 

0.566 

0.734 

0.607 

0.465 

WQ26 

0.626 

0.590 

0.661 

0.650 

0.480 

0.576 

0.531 

0.428 

0.606 

0.663 

0.698 

0.605 

0.678 

0.750 

0.697 

0.688 

0.614 

0.586 

WQ27 

0.712 

0.410 

0.501 

0.568 

0.544 

0.199 

0.741 

0.482 

0.455 

0.430 

0.356 

0.474 

0.412 

0.380 

0.349 

0.517 

0.664 

0.428 

WQ28 

0.787 

0.686 

0.718 

0.599 

0.667 

0.464 

0.648 

0.616 

0.593 

0.607 

0.566 

0.619 

0.637 

0.650 

0.594 

0.722 

0.743 

0.633 

WQ29 

0.717 

0.471 

0.516 

0.521 

0.585 

0.303 

0.599 

0.483 

0.505 

0.474 

0.426 

0.501 

0.422 

0.455 

0.374 

0.430 

0.695 

0.452 

WQ30 

0.656 

0.480 

0.531 

0.569 

0.591 

0.315 

0.600 

0.485 

0.466 

0.452 

0.426 

0.514 

0.519 

0.464 

0.457 

0.504 

0.728 

0.538 

WQ31 

0.714 

0.592 

0.523 

0.584 

0.684 

0.381 

0.714 

0.555 

0.523 

0.515 

0.457 

0.564 

0.520 

0.569 

0.514 

0.614 

0.716 

0.540 

WQ32 

0.663 

0.535 

0.466 

0.558 

0.750 

0.373 

0.706 

0.531 

0.461 

0.504 

0.495 

0.634 

0.568 

0.554 

0.548 

0.725 

0.771 

0.636 

WQ33 

0.668 

0.532 

0.473 

0.575 

0.635 

0.300 

0.696 

0.533 

0.489 

0.477 

0.371 

0.484 

0.471 

0.470 

0.470 

0.521 

0.674 

0.471 

WQ34 

0.671 

0.526 

0.485 

0.544 

0.666 

0.390 

0.639 

0.486 

0.509 

0.523 

0.510 

0.540 

0.604 

0.573 

0.568 

0.641 

0.700 

0.584 

WQ35 

0.696 

0.483 

0.538 

0.475 

0.585 

0.259 

0.611 

0.518 

0.490 

0.430 

0.358 

0.481 

0.403 

0.432 

0.402 

0.507 

0.628 

0.480 

WQ36 

0.747 

0.501 

0.530 

0.524 

0.673 

0.349 

0.630 

0.456 

0.515 

0.442 

0.468 

0.515 

0.474 

0.466 

0.484 

0.575 

0.632 

0.537 
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WQ19 

WQ20 

WQ21 

WQ22 

WQ23 

WQ24 

WQ25 

WQ26 

WQ27 

WQ28 

WQ29 

WQ30 

WQ31 

WQ32 

WQ33 

WQ34 

WQ35 

WQ36 

WQ1 

0.528 

0.695 

0.523 

0.563 

0.573 

0.400 

0.558 

0.626 

0.712 

0.787 

0.717 

0.656 

0.714 

0.663 

0.668 

0.671 

0.696 

0.747 

WQ2 

0.727 

0.831 

0.748 

0.751 

0.739 

0.614 

0.650 

0.590 

0.410 

0.686 

0.471 

0.480 

0.592 

0.535 

0.532 

0.526 

0.483 

0.501 

WQ3 

0.635 

0.699 

0.578 

0.620 

0.664 

0.579 

0.598 

0.661 

0.501 

0.718 

0.516 

0.531 

0.523 

0.466 

0.473 

0.485 

0.538 

0.530 

WQ4 

0.540 

0.571 

0.659 

0.519 

0.549 

0.433 

0.518 

0.650 

0.568 

0.599 

0.521 

0.569 

0.584 

0.558 

0.575 

0.544 

0.475 

0.524 

WQ5 

0.547 

0.605 

0.678 

0.652 

0.659 

0.333 

0.429 

0.480 

0.544 

0.667 

0.585 

0.591 

0.684 

0.750 

0.635 

0.666 

0.585 

0.673 

WQ6 

0.526 

0.634 

0.688 

0.596 

0.655 

0.695 

0.505 

0.576 

0.199 

0.464 

0.303 

0.315 

0.381 

0.373 

0.300 

0.390 

0.259 

0.349 

WQ7 

0.495 

0.496 

0.517 

0.527 

0.564 

0.306 

0.524 

0.531 

0.741 

0.648 

0.599 

0.600 

0.714 

0.706 

0.696 

0.639 

0.611 

0.630 

WQ8 

0.674 

0.696 

0.784 

0.719 

0.793 

0.511 

0.480 

0.428 

0.482 

0.616 

0.483 

0.485 

0.555 

0.531 

0.533 

0.486 

0.518 

0.456 

WQ9 

0.599 

0.689 

0.577 

0.627 

0.677 

0.592 

0.585 

0.606 

0.455 

0.593 

0.505 

0.466 

0.523 

0.461 

0.489 

0.509 

0.490 

0.515 

WQ10 

0.629 

0.739 

0.670 

0.665 

0.689 

0.613 

0.639 

0.663 

0.430 

0.607 

0.474 

0.452 

0.515 

0.504 

0.477 

0.523 

0.430 

0.442 

WQ11 

0.634 

0.744 

0.686 

0.670 

0.716 

0.654 

0.658 

0.698 

0.356 

0.566 

0.426 

0.426 

0.457 

0.495 

0.371 

0.510 

0.358 

0.468 

WQ12 

0.726 

0.717 

0.781 

0.679 

0.747 

0.389 

0.558 

0.605 

0.474 

0.619 

0.501 

0.514 

0.564 

0.634 

0.484 

0.540 

0.481 

0.515 

WQ13 

0.635 

0.664 

0.788 

0.532 

0.610 

0.583 

0.641 

0.678 

0.412 

0.637 

0.422 

0.519 

0.520 

0.568 

0.471 

0.604 

0.403 

0.474 

WQ14 

0.751 

0.734 

0.731 

0.672 

0.674 

0.722 

0.696 

0.750 

0.380 

0.650 

0.455 

0.464 

0.569 

0.554 

0.470 

0.573 

0.432 

0.466 

WQ15 

0.623 

0.621 

0.733 

0.602 

0.626 

0.566 

0.566 

0.697 

0.349 

0.594 

0.374 

0.457 

0.514 

0.548 

0.470 

0.568 

0.402 

0.484 

WQ16 

0.598 

0.645 

0.729 

0.622 

0.682 

0.555 

0.734 

0.688 

0.517 

0.722 

0.430 

0.504 

0.614 

0.725 

0.521 

0.641 

0.507 

0.575 

WQ17 

0.657 

0.652 

0.749 

0.655 

0.690 

0.339 

0.607 

0.614 

0.664 

0.743 

0.695 

0.728 

0.716 

0.771 

0.674 

0.700 

0.628 

0.632 

WQ18 

0.785 

0.699 

0.807 

0.718 

0.761 

0.457 

0.465 

0.586 

0.428 

0.633 

0.452 

0.538 

0.540 

0.636 

0.471 

0.584 

0.480 

0.537 

WQ19 

jm 

0.853 

0.770 

0.731 

0.716 

0.346 

0.446 

0.504 

0.419 

0.507 

0.523 

0.522 

0.506 

0.455 

0.441 

0.432 

0.421 

0.350 

WQ20 

0.853 

l.OftO 

0.703 

0.734 

0.695 

0.341 

0.534 

0.524 

0.430 

0.593 

0.540 

0.505 

0.538 

0.488 

0.457 

0.443 

0.464 

0.517 

WQ21 

0.770 

0.703 

|£H 

0.622 

0.683 

0.418 

0.565 

0.517 

0.339 

0.444 

0.411 

0.467 

0.478 

0.495 

0.463 

0.473 

0.289 

0.314 

WQ22 

0.731 

0.734 

0.622 

Sii 

0.889 

0.386 

0.368 

0.391 

0.339 

0.458 

0.438 

0.410 

0.455 

0.443 

0.386 

0.337 

0.361 

0.393 

WQ23 

0.716 

0.695 

0.683 

0.889 

.1.000 

0.416 

0.421 

0.376 

0.428 

0.515 

0.454 

0.460 

0.458 

0.471 

0.451 

0.415 

0.365 

0.377 

WQ24 
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M 
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0.034 
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0.023 
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0.203 

0.153 
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0.300 

0.073 
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WQ25 

0.446 

0.534 

0.565 

0.368 

0.421 

0.601 

§M 

0.765 

0.459 

0.622 

0.419 

0.440 

0.569 

0.589 

0.475 

0.583 

0.395 

0.431 

WQ26 

0.504 

0.524 

0.517 

0.391 

0.376 

0.633 

0.765 

MS 

0.372 

0.598 

0.440 

0.505 

0.530 

0.580 

0.385 

0.614 

0.416 

0.513 

WQ27 

0.419 

0.430 

0.339 

0.339 

0.428 

0.034 

0.459 

0.372 

1.00P 

0.774 

0.891 

0.737 

0.872 

0.747 

0.892 

0.709 

0.872 

0.704 

WQ28 

0.507 

0.593 

0.444 

0.458 

0.515 

0.342 

0.622 

0.598 

0.774 

'.000 
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0.824 
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WQ29 
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0.023 

0.419 

0.440 

0.891 
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Mi 
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0.706 

0.852 

0.720 

0.848 
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WQ30 

0.522 

0.505 

0.467 

0.410 

0.460 

0.119 

0.440 

0.505 

0.737 

0.763 

0.841 

|iM§ 

0.726 

0.700 

0.756 

0.845 

0.679 

0.739 

WQ31 

0.506 

0.538 

0.478 

0.455 

0.458 

0.203 

0.569 

0.530 

0.872 

0.831 

0.860 

0.726 

m 

0.879 

0.931 

0.815 

0.886 

0.769 

WQ32 

0.455 

0.488 

0.495 

0.443 

0.471 

0.153 

0.589 

0.580 

0.747 

0.824 

0.706 

0.700 
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Mfl 
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0.451 

0.140 
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0.892 

0.807 

0.852 
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I» 
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WQ34 
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0.443 

0.473 
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0.415 

0.300 
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0.709 

0.832 

0.720 

0.845 

0.815 

0.879 

0.794 

0.724 

0.835 

WQ35 

0.421 

0.464 

0.289 

0.361 

0.365 

0.073 

0.395 

0.416 

0.872 

0.797 

0.848 

0.679 

0.886 

0.775 

0.839 

0.724 
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Question 

Group 

Mean 

Sig. 

AQ1 

Jr./Sr. 

4.75/4.08 

.102 

AQ2 

Jr./Sr. 

4.53/4.11 

.388 

AQ3 

Jr./Sr. 

5.03/4.04 

.015 

AQ4 

Jr./Sr. 

5.34/4.11 

.002 

AQ5 

Jr./Sr. 

5.53/4.63 

.034 

AQ6 

Jr./Sr. 

5.66/5.11 

.183 

AQ7 

Jr./Sr. 

5.67/5.08 

.110 

AQ8 

Jr./Sr. 

5.23/4.13 

.006 

AQ9 

Jr./Sr. 

4.64/3.67 

.012 

AQ10 

Jr./Sr. 

4.85/3.70 

.010 

AQ1 1 

Jr./Sr. 

5.22/4.11 

.017 

AQ12 

Jr./Sr. 

4.47/3.54 

.022 

AQ13 

Jr./Sr. 

5.00/3.81 

.004 

AQ14 

Jr./Sr. 

4.97/4.36 

.133 

AQ15 

Jr./Sr. 

5.17/4.15 

.012 

AQ16 

Jr./Sr. 

4.88/4.19 

.089 

AQ17 

Jr./Sr. 

4.86/4.24 

.136 

AQ18 

Jr./Sr. 

4.69/3.93 

.074 

AQ25 

Jr./Sr. 

4.11/4.62 

.287 

AQ26 

Jr./Sr. 

4.28/4.44 

.743 

AQ19 

Jr./Sr. 

4.77/3.62 

.006 

AQ20 

Jr./Sr. 

4.57/3.93 

.093 

AQ21 

Jr./Sr. 

5.19/4.29 

.043 

AQ22 

Jr./Sr. 

5.44/5.16 

.513 

AQ23 

Jr./Sr. 

4.88/4.27 

.124 

AQ24 

Jr./Sr. 

4.43/4.59 

.734 

AQ27 

Jr./Sr. 

5.23/5.33 

.787 

AQ28 

Jr./Sr. 

4.83/4.35 

.244 

AQ29 

Jr./Sr. 

5.57/5.48 

.790 

AQ30 

Jr./Sr. 

5.09/4.61 

.223 

AQ31 

Jr./Sr. 

5.25/5.39 

.699 

AQ32 

Jr./Sr. 

4.94/4.48 

.262 

AQ33 

Jr./Sr. 

5.33/5.14 

.633 

AQ34 

Jr./Sr. 

4.96/4.55 

.300 

AQ35 

Jr./Sr. 

5.31/4.95 

.363 

AQ36 

Jr./Sr. 

4.89/4.41 

.242 

Question 

Group 

Mean 

Sig. 

WQ1 

Jr./Sr. 

5.03/4.50 

.080 

WQ2 

Jr./Sr. 

5.11/4.12 

.007 

WQ3 

Jr./Sr. 

4.89/4.00 

.012 

WQ4 

Jr./Sr. 

5.11/4.42 

.039 

WQ5 

Jr./Sr. 

5.11/4.58 

.151 

WQ6 

Jr./Sr. 

5.11/4.44 

.093 

WQ7 

Jr./Sr. 

5.31/5.31 

.984 

WQ8 

Jr./Sr. 

5.03/4.52 

.176 

WQ9 

Jr./Sr. 

4.75/3.91 

.020 

WQ10 

Jr./Sr. 

5.00/4.08 

.011 

WQ11 

Jr./Sr. 

5.23/4.35 

.018 

WQ12 

Jr./Sr. 

5.00/4.00 

.010 

WQ13 

Jr./Sr. 

5.14/4.00 

.003 

WQ14 

Jr./Sr. 

5.17/4.33 

.026 

WQ15 

Jr./Sr. 

5.15/3.80 

.000 

WQ16 

Jr./Sr. 

5.03/4.30 

.042 

WQ17 

Jr./Sr. 

5.06/4.35 

.047 

WQ18 

Jr./Sr. 

4.91/3.88 

.004 

WQ25 

Jr./Sr. 

4.91/4.15 

.051 

WQ26 

Jr./Sr. 

4.97/3.96 

.011 

WQ19 

Jr./Sr. 

4.67/3.73 

.015 

WQ20 

Jr./Sr. 

4.51/3.81 

.093 

WQ21 

Jr./Sr. 

5.00/4.00 

.027 

WQ22 

Jr./Sr. 

4.74/4.21 

.244 

WQ23 

Jr./Sr. 

4.46/4.34 

.799 

WQ24 

Jr./Sr. 

5.25/4.52 

.028 

WQ27 

Jr./Sr. 

5.03/5.46 

.286 

WQ28 

Jr./Sr. 

5.29/4.81 

.191 

WQ29 

Jr./Sr. 

5.27/5.43 

.698 

WQ30 

Jr./Sr. 

5.21/4.86 

.370 

WQ31 

Jr./Sr. 

5.50/5.50 

1.000 

WQ32 

Jr./Sr. 

5.38/4.80 

.173 

WQ33 

Jr./Sr. 

5.29/5.55 

.550 

WQ34 

Jr./Sr. 

5.13/4.79 

.330 

WQ35 

Jr./Sr. 

5.36/5.27 

.830 

WQ36 

Jr./Sr. 

5.09/4.65 

.265 
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WST  Needed  Features 

WST  Comments 

AMST  Needed  Features 

AMST  Comments 

Simpler  web  site 

Good  information 

Simpler  Web  site 

CPU  problems 

Pop-up  box  on  screen 
instead  of  bottom  of  screen 

Excellent  system,  easy  to 
use. 

Map,  expenses,  and  asset 
lot  number  information 

Frustrating.  Need  expert 
guidance,  not  very  fast 

A  more  user  friendly 
screen  and  hard  copy  of 
help  manual. 

Mostly  good,  a  tutorial 
would  definitely  help 
novice  users 

Condensed  help  feature, 
too  in-depth.  Less 
information  input  more 
point  and  click 

Excellent!  Inform  all  Bases 
of  systems  ASAP  as  a  first 
time  user,  it  was  relatively 
easy  to  use  and  helpful 

JHCS,  ERRC,  ADR,  RDO, 
CRD,  and  RIMF 

Had  problems  with  aircraft 
compatibility  feature 

Additional  security 
features 

Easy  to  navigate.  In  transit 
option  is  great 

Need  training  class  (1  day) 
and  print  capabilities 

Great  system,  Good  source 
of  munitions  information 

Fed  Log  information  and 
AMMO  catalog 

Good  resource  tool. 
Concerned  with  security. 

COTS  Approval, 
Forecasting,  and  allocation 
breakdown 

Excellent!  Novice  use! 
Help  messages  great  and 
easy  to  use 

Weapons  profile  and 
definition  of  terms. 

Great  tool  but  concerned 
with  security  of  quantities, 
could  be  a  threat  to  OPSEC 

Indicative  Data, 
Expenditure,  and 
requisitions  information 

Good  product;  however, 
some  of  the  pop-up  menu 
descriptions  hide  behind 
the  toolbar.  Hard  to  read. 

Quick  help  like  in  Fed 
Log  and  blow  up  of 
states  in  Asset  Map  for 
easier  use. 

Would  make  everything 
easier  if  we  could  get 
everything 

Incorporate  with  AMST 

Not  as  good  as  AMST 

Master  NSN  Listings 

Difficult 

Clear  titles,  NSNs  and 
reference  to  applicable 
TOs 

Enjoyable,  very  user 
friendly  system  but  very 
mission  specific 

Acronym  explanation 
and  help  in  query 

Good  System  for  depot 
assets.  Need  to  know  what 
you  are  looking  for. 

Attention  to  Security 

Good  fast  information 

Hard  to  use  for  first  timers 

Better  layout  of  features 

Informative 

Overall  good  program 

Excellent  user  interface 
but  lacks  depth  of 
information.  No 
information  pertaining  to 
Fed  Log/HAZDEC 

Older  CPU  caused 
frustrating  experience. 
Unable  to  get  information 
fast  if  at  all. 

Weapon  diagrams  nice 
feature.  Term  breakdown 
helpful.  Difficult  to  find 
specific  munitions  without 
the  specific  number. 

Interesting  with  a  lot  of 
potential  but  information 
may  be  dangerous.  Can  be 
helpful  to  a  manager  in  the 
field.  “ 

Frustrating  due  to  old 
computer,  menu 
navigation  not  friendly 

Frustrating,  have  to  enter 
NSNs  etc.  to  get 
information 

Difficult  to  move  around. 
Numerous  problems  with 
navigating  system.  Need 
more  time  to  learn  system 

Very  easy  to  navigate 
however,  a  manual  should 
be  included  for  beginners 
and  non-munitions  users. 
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14.  ABSTRACT 

The  purpose  of  this  research  is  to  ascertain  if  the  United  States  Air  Force  (USAF)  is  duplicating  effort  with  the  development  of  the  two  IT 
applications,  or  if  one  will  effectively  meet  requirements.  Specifically,  this  thesis  sought  to  answer  four  research  questions  addressing  customer 
needs  for  a  munitions  IT  application,  identifying  the  capabilities  of  the  two  research  IT  applications,  detennining  how  well  each  application  met  the 
needs  of  the  customer,  and  detennining  how  supportable  each  application  was  in  terms  of  information  needs  versus  existing  munitions  systems.  A 
mixed  qualitative  and  quantitative  paradigm  was  used  to  conduct  a  gap  analysis  and  verification  occurred  through  the  use  of  surveys.  Sixty-five 
munitions  users  participated  in  a  surveying  session  to  identify  differences  between  the  two  systems  as  well  as  the  relative  merits  of  each.  Overall, 
munitions  users  chose  WST  as  the  more  favorable  application.  All  but  two  construct  measures  supported  the  overall  rating  of  the  application. 
Munitions  personnel  liked  the  screen  layout  of  AMST  better  than  that  of  WST  even  though  WST  seemed  to  be  the  application  that  better  used 
terminology  familiar  to  munitions  personnel.  WST  was  also  the  easier  of  the  two  applications  to  manipulate  although  the  AMST  help  features  were 
identified  as  the  better  of  the  two.  The  Warrior  Support  Tool  provided  information  in  a  timelier  manner  than  did  the  Agile  Munitions  Support  Tool. 
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